(19) 



(12) 



/( Europaisches Patentamt 
European Patent Office 
Office europtendes brevets (11) EP 0 913 790 A1 

EUROPEAN PATENT APPLICATION 

(51) Int. Ci.»: G06K 11/08, G06F 3/00 



(43) Date of publication: 

06.05.1999 Bulletin 1999/18 

(21) Application numt)er: 98120405.0 

(22) Date of filing: 28.10.1998 



(84) Designated Contracting States: 


(72) Inventor: 


AT BE CH CY DE DK ES R FR GB GR IE IT LI LU 


Harakawa, Kenlchi, 


MCNLPTSE 


Technical Lab. Takenaka Corp. 


Designated Extension States: 


Inzal-shI, Chiba-ken (JP) 


AL LT LV MK RO SI 






(74) Representative: 


(30) Priority: 29.10.1997 JP 296788/97 


Klunker . Schmitt-Nllson . Hirsch 




Winzererstrasse 106 


(71 ) Applicant: TAKEN AKA CORPORATION 


80797 MQnchen (DE) 


Osaka-shI, Osaka, 541-0053 (JP) 





(54) Hand pointing apparatus 

(57) An image which represents a S-D space virtu- 
ally using a perspective method is displayed on a dis- 
play, and an image of an information inputting person is 
picked up by a plurality of video cameras from different 
directions. While the information inputting person is 
pointing at an arbitrary position within a virtual 3-D 
space represented by the displayed image, a reference 
point which corresponds to the back of the information 
inputting person and a characteristic point which corre- 
sponds to a finger tip are respectively extracted from a 
plurality of images which are picked up by each of the 
video cameras, and the 3-D coordinates of these points 
are determined. The direction in which the position 
pointed to by the information inputting person is dis- 
posed within the virtual 3-D space is determined on the 
basis of the direction from the reference point to the 
characteristic point, and tiie distance between the infor- 
mation inputting person and the position pointed to by 
the information inputting person within tiie virtual 3-D 
space is determined on the basis of the distance 
between the reference point and the characteristic 
point. As a result, the 3-D coordinates of ttie position 
pointed to by the information inputting person within the 
virtual 3-D space can be determined. 
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Description 

BACKGROUND OF THE INVENTION 

Field of the Invention s 

[0001] The present invention relates to a hand point- 
ing apparatus, and more specifically to a hand pointing 
apparatus for picking up a person to be recognized from 
a plurality of different directions and for determining the io 
coordinates of a specific position pointed to by the per- 
son to be recognized. 

Description of the Related Art 

15 

[0002] There has been heretofore known a hand 
pointing input apparatus which comprise a display for 
displaying predetermined information, an illumination 
device for illuminating an information inputting person 
who comes to the display, and a plurality of image 20 
pickup means for picking up the image of the approach- 
ing information inputting person from different direc- 
tions, wherein a plurality of image pickup means image 
pickup images of situations where the approaching 
information inputting person points with a finger or the 25 
like to an optional position on the display, the informa- 
tion inputting person is recognized in accordance with a 
plurality of images obtained by the image pickup, the 
position on the display pointed to by the information 
inputting person is determined, a cursor or the like is 30 
displayed on the position pointed to on the display, and 
the position on the display pointed to is recognized as 
being clicked at the time of detecting the fact that the 
information inputting person has performed a clicking 
action by raising a thumb, whereby a predetermined 35 
processing is performed (see, for example. Japanese 
Patent Application {.aid-Open (JP-A) Nos. 4-271423, 5- 
19957, and 5-324181 or the like). 
[0003] According to the above-described hand point- 
ing input apparatus, since tiie information inputting per- 40 
son can give various instructions to an information 
processing apparatus and input various information to 
the information processing apparatus without touching 
an input device such as a keyboard or a mouse, it is 
possible to simplify the operation for using the informa- 4s 
Won processing apparatus. 

[0004] The above-described hand pointing input 
apparatus can determine which position on the display 
screen of ttie display the information inputting person is 
pointing to However, if the displayed image on the dis- 50 
play is an image spuriously representing a 3-D (tiiree- 
dimensional) space, this hand pointing input apparatus 
cannot determine which position witiiin a virtual 3-D 
space represented by the image (i.e.. the 3-D coordi- 
nates of a position pointed to by the information input- 55 
ting person) the information inputting person is pointing 
to As to spuriously displaying an image representing a 
3-D space, apart from displaying images in conformity 



with a one-point perspective or two-point perspective on 
a planar display, various methods have been also pro- 
vided in which images can be displayed on a 3-D dis- 
play using a liquid-crystal shutter or a lenticular lens, 
stereographic images can be displayed by applying 
holographic technology, and tiie like (tiiese images are 
referred to as three-dimensional images hereinafter). 
However, tiiere has been a drawback in that three- 
dimensional images such as those described above 
cannot be used as an object pointed to by tiie informa- 
tion inputting person. 

[0005] Furtiier, tiie above-described drawback is not 
limited to the case in which the object pointed to by the 
information inputting person is a virtual 3-D space rep- 
resented by a three-dimensional in^ge. It is also impos- 
sible to determine the position witiiin a 3-D space the 
information inputting person is pointing to, even when 
the object pointed to by ttie information inputting person 
exists within an actual 3-D space. 

SUMMARY OF THE INVENTION , 

[0006] In view of ttie aforementioned, it is an object of 
tiie present invention to provide a hand pointing appara- 
tus which can determine 3-D coordinates of a position 
which is pointed to by an information inputting person 
even when the person is pointing to an arbitrary position 
witiiin a 3-D space. 

[0007] In order to accomplish ttie aforementioned 
object, ttie first aspect of the present invention is a hand 
pointing apparatus, comprising: image pickup means 
which picks up ttie image of a person to be recognized 
from a plurati^ of different directions; computing means 
which exti'acts an image which corresponds to the per- 
son to be recognized on the basis of a plurality of 
images obtained by tiie image pickup means which 
picks up images from a plurality of direction of the per- 
son to be recognized pointing to a specific position 
witiiin 3-D coordinates, and determines the 3-D coordi- 
nates of a characteristic point whose position may be 
changed by the person to be recognized bending or 
stretching an arm, and a reference point whose position 
does not change even when tiie person to be recog- 
nized bends or stretches ttie arm; and determining 
means for determining tiie direction in which the specific 
position exists witiiin ttie 3-D space on tiie basis of the 
direction from ttie reference point to the characteristic 
point and for determining the location of the specific 
position witiiin the 3-D space along ttie deptti direction 
ttiereof on ttie basis of the distance between ttie refer- 
ence point and the characteristic point, and tiiereby 
determines the 3-D coordinates of the specific position 
witiiin the 3-D space. 

[0008] In the first aspect of the present invention, an 
image of the person to be recognized (the information 
inputting person) is picked up by ttie image pickup 
means, from a plurality of different directions. The 
image pickup means may be structured so that the 
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image of the person to be recognized Is picked up from 
a plurality of directions using a plurality of image pickup 
apparatuses which are comprised of video cameras or 
\the like. It can also be structured such that light reflect- 
ing means such as a plane mirror or the like is provided 
at the image pickup means, and an image of the person 
to be recognized Is picked up directly by a single image 
pickup apparatus, and the image of the person to be 
recognized is picked up from a plurality of directions by 
picking up virtual images of the person to be recognized 
which are projected onto the plane min-or. 
[0009] Further, the computing means extracts an 
image portion which corresponds to the person to be 
recognized on the basis of a plurality of images picked 
up from a plurality of directions using the Image pickup 
means, wherein the person to be recognized is pointing 
to a specific position within a 3-D space, and deter- 
mines the 3-D coordinates of a characteristic point 
whose position may be changed by the person to be 
recognized bending or stretching an arm, and a refer- 
ence point whose position does not change even when 
the person to be recognized bends or stretches an arm. 
For example, a point which corresponds to the tip of the 
hand, finger or the like of the person to be recognized or 
the tip of a pointing apparatus which is grasped by the 
person to be recognized can be used as a characteristic 
point. A point which con-esponds to the body of the per- 
son to be recognized (e.g., the breast or the shoulder 
joint of the person to be recognized) can be used as a 
reference point. The 3-D space may be a virtual 3-D 
space represented by a three dimensional image such 
as an image which is formed in conformity with a one- 
point perspective method or two-point perspective 
method on a planar display, an image which uses a liq- 
uid aystal shutter or a lenticular lens on a 3-D display, 
or a stereographic image which is displayed by applying 
holographic technology, or the 3-D space may be an 
actual 3-D space. 

[0010] The determining means for determining the 
direction in which the specific position exists within the 
3-D space, on the basis of tiie direction from the refer- 
ence point to tiie characteristic point, determines the 
position of the specific position within tiie 3-D space 
along the deptii direction thereof on tiie basis of tiie dis- 
tance between the reference point and tiie characteris- 
tic point, and thereby determines 3'D coordinates of the 
specific position witiiin tiie 3-D space. 
[0011] Accordingly, tiie person to be recognized car- 
ries out tiie operation for adjusting tiie direction of the 
characteristic point (i.e., the operation of pointing tiie 
hand, finger, or tip of a pointing device of tiie person to 
be recognized towards a specific position) with respect 
to tiie reference point such ttiat tiie direction from the 
reference point to the characteristic point con^esponds 
to the direction in which the specific position exists as 
the object pointed to. as seen from tiie point of view of 
the person to be recognized. Additionally, the person to 
be recognized cames out the operation for adjusting tiie 



distance between tiie reference point and the character- 
istic point (i.e.. the operation of bending or sti'etching the 
arm by the person to be recognized) in accordance with 
tiie distance between the specific position and the per- 

5 son to be recognized (i.e.. how near to or far from tiie 
person to be recognized) so that the direction In which 
tiie specific position exists witiiin the 3-D space and the 
position of tiie specific position exists within the 3-D 
space along the deptii direction ttiereof can be deter- 

10 mined, and tiie 3-D coordinates of the specific position 
can be determined witiiin ttie 3-D space on tiie basis of 
tiie results of the determining of tiie aforementioned 
direction and tiie position of the specific position in tiie 
depth direction thereof. 

15 [0012] In accordance with tiie first aspect of tiie 
present invention, when the Information Inputting per- 
son (the person to be recognized) points to an arbitrary 
position within the 3-D space, tiie 3-D coordinates of tiie 
position pointed to can be determined. Furttier. ttie 

20 action of pointing a hand or finger or the tip of a pointing 
device towards the direction where the specific position 
exists as seen by the person to be recognized and 
bending or extending an arm to cover tiie distance to 
the specified position by the person to be recognized is 

25 an extremely natural action for pointing to a specific 
position inside a 3-D space. Accordingly, the information 
inputting person (tiie person to be recognized) can 
carry out tiie above-described action without any 
annoyance involved with this action. 

30 [0013] As described in tiie second aspect of tiie 
present invention, the determination of tiie location of 
tiie specific position along tiie deptii direction of tiie 3-D 
space on the basis of the distance between the refer- 
ence point and tiie characteristic point can be effected 

35 by converting the distance between tiie reference point 
and the characteristic point into tiie distance between 
the person to be recognized and tiie specific position 
according to a predetermined conversion conditions. 
The aforementioned conversion conditions can be con- 

40 version characteristics in which the distance between 
tiie person to be recognized and the specific position 
may vary linearly or non-llnearly to correspond to tiie 
change of tiie distance between tiie reference point and 
tiie characteristic point. 

45 [0014] Especially when tiie 3-D space serving as an 
object which is pointed to by the person to be recog- 
nized is a space whose deptii is very long (e.g.. when a 
tiiree dimensional Image which represents tiie universe 
is displayed on the display means), if the conversion 

so characteristics of ttie conversion conditions are made 
non-linear (conversion characteristics or tiie like which 
cause tiie distance between the person to be recog- 
nized and the specific position to vary in proportion to 
the number raised to tiie ntii (n g 2) power of tiie 

55 change in the distance between tiie reference point and 
the characteristic point), it allows the person to be rec- 
ognized to point to a position located at an extreme dis- 
tance witiiin tiie 3-D space as seen from tiie person to 
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be recognized without carrying out exaggerated actions 
sucli as ,the stretching or bending of an arm beyond 
what is normal. As a result, it is preferable because the 
person to be recognized can be prevented from being 
burdened by the action of pointing to an arbitrary posi- 5 
tion within a 3-D space. 

[001 5] Further, because the physique of the person to 

be recognized (especially, the length of the arm) is not 
fixed, the width of the movement of the characteristic 
point when the person to be recognized bends or 10 
stretches an arm is different for each person. When the 
distance between the reference point and the character- 
istic point is converted into the distance between the 
person to be recognized and the specific position in 
accordance with fixed conversion conditions, the case is 
in which the location of the specific position which is 
pointed to by the person to be recognized cannot be 
determined accurately due to variables such as the indi- 
vidual lengths of the arms of the persons to be recog- 
nized or the like can be thought of.. . ' 20 
[0016] For this reason, in accordance with the third 
aspect of the present invention, there is provided a hand 
pointing apparatus according to the second aspect of 
the present invention, further comprising: conversion 
conditions setting means which requests the person to 25 
be recognized to cany out the arm bending or stretching 
acton, and sets in advance the conversion conditions 
which convert the distance between said reference 
point and the characteristic point into the distance 
between the person to be recognized and the specific so 
position on the basis of the extent of the change in the 
distance between the reference point and the character- 
istic point when the person to be recognized cames out 
the arm bending or stretching action. 
[0017] According to the third aspect of the present 35 
invention, since the conversion conditions which con- 
vert the distance between the reference point and the 
characteristic point into the distance between the per- 
son to be recognized and the specific position is set in 
advance on the basis of the extent of the change in the 40 
distance between the reference point and the character- 
istic point (the extent of the change in the distance may 
vary due to the individual lengths of the arms of the per- 
sons to be recognized or the like) when the person to be 
recognized carries out the arm bending or stretching 45 
action, conversion conditions can be obtained in 
accordance with the physique of each of the person to 
be recognized. By candying out the aforementioned con- 
version using these conversion conditions, in spite of 
the variables arising from the individual physiques of the so 
persons to be recognized, it is possible to accurately 
determine the location of the specific position along the 
depth direction thereof which Is pointed to by the person 
to be recognized. Moreover, for example, in order to 
inform the hand pointing apparatus of the fact that the ss 
person to be recognized is pointing to an extremely 
remote position within the 3-D space, the person to be 
recognized whose physique is especially small does not 



need to carry out any exaggerated actions beyond what 

is normal. 

[0018] In accordance with the third aspect of the 
present invention, a single position which is located in 
the intermediate portion of a 3-D space along the deptii 
direction thereof, or a plurality of portions whose loca- 
tions are different from each otiier within ttie 3-D space 
in the depth direction thereof (the location of position in 
the 3-D space along the deptti direction thereof is 
already known) is pointed to by tiie person to be recog- 
nized so ttiat conversion conditions (tiie configuration of 
a conversion curve) can be set on the basis of the dis- 
tance between tiie reference point and the characteris- 
tic point at tills time. 

[0019] In accordance with tiie Iburtii aspect of tiie 
present invention, there is provided a hand pointing 
apparatus according to the first aspect of the present 
invention, further comprising: display means which dis- 
plays a three dimensional image; display control means 
which displays tiie three dimensional image on the dis- 
play means; and determining means which deternriines 
whether tiie hand of the person to be recognized js in a 
specific shape, wherein the three dimensional image is 
an image which represents a virtual 3-D space, and 
includes an image which is formed confirming to one- 
point perspective metiiod or two-point perspective 
metiiod, an image which uses a liquid crystal shutter or 
a lenticular lens, and a stereographic image which is 
displayed by applying holographic technology, and tiie 
person to be recognized points to a specific position 
within the virtual 3-D space which is represented by the 
three dimensional image which is displayed on the dis- 
play means, and in a state in which tiie hand of the per- 
son to be recognized is determined to be in a specific 
shape, when the distance between tiie reference point 
and the characteristic point changes, tiie display control 
means controls the display means such tiiat tiie ttiree 
dimensional image displayed on tiie display means is 
displayed so as to be enlarged or reduced according to 
the change in tiie distance between tiie reference point 
and tiie characteristic point 

[0020] The fburtii aspect of tiie present invention is 
structured such tiiat a tiiree dimensional image is dis- 
played on tiie display means, and the person to be rec- 
ognized points to the specific position within a virtual 3- 
D space which is represented by the three dimensional 
image. A plane display for displaying an image which is 
formed conforming to a one-point perspective metiiod 
or two-point perspective method, a 3-D display which 
uses a liquid crystal shutter or a lenticular lens, and a 
display apparatus which displays a stereographic image 
formed through holographic technology can be used for 
tiie display means. As described above, when a ttiree 
dimensional image is displayed on tiie display means, 
because tiie display magnification can be set arbitrarily, 
it is preferable that the display magnification can be 
changed by an inst'uction from the person to be recog- 
nized. However, it is necessary to distinguish the action 
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in which the person to be recognized changes the dis- 
play niagnlf icat'on from the action in which the person to 
be recognized points to the specific position within the 
\3-D space. 

[0021] In the fourth aspect of the present invention, 
there is provided determining means which determines 
whether the hand of the person to be recognized is in a 
specific shape. In a state in which the hand of the per- 
son to be recognized is determined to be in a specific 
shape, when the distance between the reference point 
and the characteristic point has changed, the display 
control means controls the display means such that the 
three dimensional image displayed on the display 
means is displayed so as to be enlarged or reduced 
according to the change in the distance between the ref- 
erence point and the characteristic point. It is desirable 
that the aforementioned specific shape be easily deter- 
mined. For example, a hand shape in a state in which 
the fingers are stretched out so as to open the hand can 
be used. 

[0022] Accordingly, in a state in which the hand of the 
person to be recognized is in a specific shape (a state in 
which the fingers are stretched out so as to open the 
hand), when the action of changing the distance 
between the reference point and the characteristic point 
(the bending or stretching action of the arm of the per- 
son to be recognized) is earned out, this action is distin- 
guished from the action of pointing to the specific 
position, and the display magnification of a three dimen- 
sional image which is displayed on the display means is 
changed due to the change in the distance between the 
reference point and the characteristic point so as to be 
enlarged or reduced. As a result, in accordance with the 
fourth aspect of the present invention, the display mag- 
nification of the three dimensional image displayed on 
the display means can be reliably changed due to the 
variation of the display magnification of the three dimen- 
sional image which is instructed by the person to be rec- 
ognized. 

[0023] The aforementioned display magnification can 
be changed linearly or non-linearly in response to the 
change in the distance between the reference point and 
the characteristic point. 

[0024] As described above, in the structure in which 
the 3-D coordinates of the specific position within the 3- 
D space, which is pointed to by the person to be recog- 
nized, preferably, a predetermined process can be exe- 
cuted by the person to be recognized carrying out a 
specified action (a so-called click action). However, 
depending upon the direction of the image pickup by the 
image pickup means, the action of lifting a thumb which 
has been conventionally adopted as the click action 
cannot be detected. Further, the degree of freedom in 
the action of lifting a thumb is very limited as an action. 
It is also difficult to provide a plurality of meanings for 
the click action in the same manner as a right click or a 
left click on a mouse and select a process which is exe- 
cuted by a click action. 



[0025] For this reason, in accordance with the fifth 
aspect of the present invention, there is provided a hand 
pointing apparatus according to the first aspect of the 
present invention, further comprising: processing 

5 means which detects the speed at which the distance 
between the reference point and the characteristic point 
changes, and executes a predetermined process when 
the detected speed at which the distance between the 
reference point and the characteristic point changes is 

10 greater than or equal to a threshokJ value. 

[0026] In the fifth aspect of the present invention, if the 
person to be recognized canries out a quick arm bend- 
ing or stretching action, the distance between the refer- 
ence point and the characteristic point changes at a 

75 speed which is greater than or equal to a threshold 
value. By using this changing speed as a trigger, a pre- 
determined process is carried out. Moreover, the 
processing means can execute a process which is rele- 
vant to the specific position which is pointed by the per- 

20 son to be recogriized when the distance between the 
reference point and the characteristic point has 
changed at the speed which is greiater than or equal to 
the threshold value. 

[0027] The present invention determines the 3-D coor- 
25 dinates of the specific position which is pointed to by the 
person to be recognized on the basis of the positional 
relationship between the reference point and the char- 
acteristic point. However, in accordance with the fifth 
aspect of the present invention, because it Is deter- 
so mined whether a predetermined process has been 
instructed to be executed on the basis of the diange of 
each of the positions of the reference point and the 
characteristic point, the image pickup direction by the 
image pickup means can be fixed so that the reference 
35 point and the characteristic point can reliably be 
detected without considering the lifting or lowering 
action of a finger. As a result, in accordance with the fifth 
aspect of the present invention, the action by which a 
predetermined process is instructed by the person to be 
40 recognized (the action of quickly bending or stretching 
the arm of the person to be recognized) can be detected 
reliably 

[0028] There are two types of directions in which the 
distance between the reference point and the character- 
's istic point changes (the direction in which the distance 
increases and the direction in which the distance 
decreases). Accordingly, as described in the sixth 
aspect of the present invention, when the distance 
between the reference point and the characteristic point 
so increases at the speed of the change which is greater 
than or equal to the threshold value, the first predeter- 
mined process can be executed, and when the distance 
between the reference point and the characteristic point 
decreases at the speed of the change which is greater 
55 than or equal to the threshold value, the second prede- 
termined process which is different from the first prede- 
termined process can be executed. 
[0029] In the sixth aspect of the present invention, the 
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first predetermined process Is executed when the per- 
son to be recognized carries out the quick action of 
stretching an arm (in this case, the distance between 
thd^ reference point and the characteristic point 
increases at the changing speed which is greater than 5 
or equal to the threshold value). The second predeter- 
mined process is executed when the person to be rec- 
ognized carries out a quick action of bending an arm (in 
this case, when the distance between the reference 
point and the characteristic point decreases at the 
changing speed which is greater than or equal to the 
threshold value). Accordingly, it becomes possible for 
the person to be recognized to select one of the first and 
the second processes as in the right dick and the left 
click on the mouse. By performing one of the aforemen- 
tioned actions, it is possible to reliably execute a proc- 
ess selected from the first predetermined process and 
the second predetermined process by the person to be 
recognized. 

[0030] As described above, because the physique of 
the person to be recognized is not universal, and 
accordingly, the muscular strength or the like of the per- 
son to be recognized is not universal, even when the 
person to be recognized carries out a quick arm bend- 
ing or stretching action in order to cause the processing 
means to execute a predetermined process, the speed 
at which the distance between the reference point and 
the characteristic point changes is different for every 
individual person to be recognized. As a result, even if 
the person to be recognized carries out a quick arm 
bending or stretching action in order to cause the 
processing means to execute a predetermined process, 
the bending or stretching action cannot be detected. 
Instead, it may even be possible that the aforemen- 
tioned action is detected although the person to be rec- 
ognized has not carried out this action. 
[0031] For this reason, the seventh aspect of the 
present invention is a hand pointing apparatus accord- 
ing to the fifth aspect of the present invention, further 
comprising: threshold value setting means which 
requests the person to be recognized to cany out the 
arm bending or stretching action to cause the process- 
ing means to execute the predetermined process, and 
thereby sets the threshold value in advance on the basis 
of the speed at which the distance between the refer- 
ence point and the characteristic point changes when 
the person to be recognized canries out the arm bending 
or stretching action. 

[0032] In the seventh aspect of the present invention, 
because a threshold value for determining whether the 
processing means executes a predetermined process is 
set in advance on the basis of the speed at which the 
distance between the reference point and the character- 
istic point changes when the person to be recognized 
carries out the arm bending or stretching action in order 
to execute a predetermined process by the processing 
means, a threshold value according to the physique, the 
muscular strength or the like for each person to be rec- 



ognized can be provided. By determining whether a pre- 
determined process is Instructed to be executed on the 
basis of the threshold value, in spite of variations in the 
physique, the muscular strength or the like for each per- 
son to be recognized, it can be reliably detected that the 
person to be recognized has given an instruction to exe- 
cute a predetermined process. As a result, a predeter- 
mined process can be executed reliably. 



10 BRIEF DESCRIPTION OF THE DRAWINGS 
[0033] 

Rg. 1 is a perspective view which illustrates the 
15 area sunounding an information inputting space. 

Rg. 2 Is a block diagram which illustrates a sche- 
matic structure of a hand pointing input apparatus 
according to an embodiment of the present inven- 
tion. 

20 Rg. 3 is a schematic view illustrating an example of 
the relationship between the illumination range of 
an illumination apparatus and the Image pickup 
range of a video camera. 

Rg. 4 Is a perspective view of the Information Input- 
25 ting space which Illustrates an example of a mark- 
ing plate. 

Rgs. 5A through 5C are flowcharts Illustrating the 
initial setting process of the grkl point positional 
information. 

30 Rgs. 6A through 6D are flowcharts Illustrating the 
instruction determination process. 
Rgs. 7A through 7C are flowcharts illustrating the 
coordination computing process of a reference 
point/characteristic point. 
35 Rgs. 8A through 8C are flowcharts illustrating the 
setting process of the distance conversion condi- 
tions. 

Rg. 9 is a flowchart which illustrates the setting 
process of a click action speed. 
40 Rg. 1 0A is a timing chart which illustrates the 
switching on/off of an illumination apparatus A. 
Rg. 1 0B is a timing chart which Illustrates the 
switching on/off of an illumination apparatus B. 
Rg. 10c is a timing chart which illustrates tiie tim- 
45 ing of the outputting (capturing) of Images obtained 
by the image pickup of a video camera. 
Rg. 11 is a side view of the information inputting 
space for illustrating tiie computation of tiie height 
of an information inputting person and the position 
50 of the person on the floor. 

Rgs. 12A through 12C are image views illustrating 
an example of tiie motion of the information input- 
ting person. 

Rg. 13A is an image view illustrating an image of a 
55 hand of the information inputting person which is 
picked up by a video camera. 
Rg. 138 is a schematic view which illustrates ttie 
retrieval range of grid points for determining tiie 
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coordinates of a characteristic point and the 3-D 
coordinates of the characteristic point. 
Fig. 14A is an image view of the motion of the infor- 
mation inputting person when the position pointed 
to is located at a long distance from the person. 
Fig. 14B is an image view of the motion of the infor- 
mation inputting person when the position pointed 
to is located at an intermediate distance from the 
person. 

Fig. 14C is an image view of the motion of the infor- 
mation inputting person when the position pointed 
to is located at a dose distance from the person. 
Fig. 15 is a diagram which illustrates an example of 
the distance conversion conditions for converting a 
distance k between the reference point and the 
characteristic point of the information inputting per- 
son into a distance L (a virtual distance) between 
the information inputting person and the position 
pointed to by the information inputting person. 
Fig. 16A is an image view illustrating a fonvard click 
action. , 
Fig. 16B is an image view illustrating a backward 
click action. 

Fig. 17A is a plan view of the information inputting 
space illustrating the determination of the position 
on the display to which the information inputting 
person is pointing. 

Fig. 17B is a side view of the information inputting 
space illustrating the determination of the position 
on the display to which the information inputting 
person is pointing. 

Fig. 18 is a schematic view illustrating the conver- 
sion from the distance k between the reference 
point and the characteristic point of the information 
inputting person into the distance L (a virtual dis- 
tance) between the information inputting person 
and the position which is pointed to by the informa- 
tion inputting person. 

Fig. 19A is an image view illustrating an example of 
a shape of a hand of the information inputting per- 
son when the person performs an action to point to 
a specific position on flie display screen. 
Fig. 198 is an image view which illustrates an 
example of a shape of a hand of the information 
inputting person when the person performs an 
action to change the display magnification of a 
three-dimensional image. 

Fig. 20A is an image view which illustrates an 

example of a three-dimensional image before the 

display magnification is changed. 

Fig. 20B is an image view whidi illusti^ates an 

example of a three-dimensional image which is 

enlarged and displayed in response to the action of 

the information inputting person. 

Fig. 20C is an image view which illustrates an 

example of the ttiree-dimensional image which is 

reduced and displayed in response to the action of 

the information inputting person. 



DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

[0034] With reference to the appended drawings, an 

5 example of an embodiment of ttie present invention will 
be described in detail hereinafter. As shown in Fig. 1 . a 
large-screen display 12 which corresponds to a display 
means according to a fourth aspect of the present 
invention is built into a wall surface in a place where an 

10 information inputting person 1 0, who is the person to be 
recognized of tiie present invention, arrives. 
[0035] The display 12 is used to display a three- 
dimensional image which virtually represents a three- 
dimensional space (which is referred to as a 3-D space 

15 hereinafter) in accordance with one-point perspective or 
two-point perspective. Well-known display apparatuses 
such as a liquid crystal display (LCD), a plasma display, 
a cathode ray tube (CRT), an optical fiber display and 
tiie like can be applied as tiie display 12. Further. 

20 instead of tiie aforementioned display 12, a 3-D display 
using a liquid crystal shutter or a lenticular lens, or a dis- 
play apparatus which displays a stereographic image by 
applying holographic technology may be used. Moreo- 
ver, crystal goggles or the like can be applied to tiie dis- 

25 play 

[0036] The display 12 is connected to an information 
processing apparatus 14 comprising a personal conrtpu- 
ter or ttie like (see Fig. 2). A tiiree<iimensional image is 
displayed on tiie display saeen, and otiier information 

30 such as a message or tiie like is also displayed on tiie 
screen overlapping the tiiree-dimensional image by tiie 
information processing apparatus 14. In the present 
embodiment, the information inputting person 10 anrives 
at a space in front of ttie display 12 which is shown in 

35 Fig. 1 (information inputting space). The information 
inputting person 10 gives various instructions to tiie 
information processing apparatus 14 and allows tiie 
information processing apparatus 14 to execute various 
processes by pointing to an arbitrary position within a 

40 virtual 3-D space represented by a three-dimensional 
image which is displayed on the display 12 and by per- 
forming a click action (of which details will be described 
later). 

[0037] As shown in Fig. 2, a contiroller 22 of a hand 
45 pointing input apparatus 20 according to the present 
embodiment is connected to tiie information processing 
apparatus 14. The controller 22 comprises a CPU 22 A, 
ROM 22B, RAM 22C, and an I/O interface 22D, which 
are connected to each otiier via a bus. The Information 
50 processing apparatus 14 is connected to the I/O inter- 
face 22D. A non-volatile storage device 24 whose stored 
contents are rewritable, a display 26 for displaying vari- 
ous types of information, a keyk)oard 28 ttirough which 
an operator inputs various instructiorre or data, an illumi- 
55 nation control apparatus 30, an image pickup control 
apparatus 34, and a marking plate driving apparatus 38 
are also connected to the I/O interface 22D. 
[0038] A large number of tiiree-dimensional images 
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which are displayed on the display 12 are prepared in 

advance, jand a large number of image data for repre- 
senting the various three<iimensional images is stored 
in the storage device 24. Further, coordinate data for 
representing the relationship between the 3-D coordi- 
nates of each of the positions within a virtual 3-D space 
represented by each of the three-dimensional images 
and the position (2D coordinates) of each of the posi- 
tions on the three-dimensional image is also stored in 
the storage device 24. Moreover, instead of the coordi- 
nate data, the aforementioned relationship may be 
stored in the form of a functional equation or the like 
(preferably, it includes a display magnification as a 
parameter). 

[0039] A plurality of near-infrared light illumination 
apparatuses 32A and 32B. which emit beams of a 
wavelength in the near-infrared range, are connected to 
the illumination control apparatus 30. As shown in Fig. 
1, the near-infrared light illumination apparatuses 32A 
and 32B are disposed in separated upper portions of 
the information input space. The illumination ranges of 
the near-infrared light illumination apparatuses 32A and 
32B are controlled such that the information inputting 
person 1 0 who anrives in the information inputttng space 
can be illuminated from different directions. The illumi- 
nation control apparatus 30 controls the switching of the 
illumination apparatuses 32A and 328. 
[0040] A plurality of video cameras 36A and 36B 
which are disposed in separated upper portions of the 
information inputting space (see Fig. 1) are connected 
to the image pickup control apparatus 34. The plurality 
of the video cameras 36A and 36B have an area sensor 
which comprises a CCD (not shown) or the like which is 
sensitive to near-infrared light, and which has a filter 
which transmits only light of a wavelength in the near- 
infrared range on a light incident side of an image form- 
ing lens which makes incident light form an image on 
the receptor surface of the area sensor. 
[0041 ] As shown in Fig. 3, the orientations of the video 
cameras 36A and 36B are adjusted so as to pick up 
images of the information inputting person 10 from dif- 
ferent directions. Namely, the orientation (the image 
pickup range) of the video camera 36A is controlled in 
such a manner that the information inputting person 10 
who amves in the information inputting space is within 
the image pickup range, and the light emitted from the 
illumination apparatus 32A is not directly incident on the 
image forming lens. Moreover, the central portion of the 
image pickup range intersects the central portion of the 
illumination range of the illumination apparatus 32A 
within the information inputting space at a predeter- 
mined height from the floor surface (the illumination 
range on the floor surface illuminated by the illumination 
apparatus 32A is out of the image pickup range). Simi- 
larly, the orientation (the image pickup range) of the 
video camera 36B is contirolled in such a manner that 
the information inputting person 10 who arrives in the 
information who arrives in the information inputting 



space is within the image pickup range, and the light 
emitted from the illumination apparatus 32B is not 
directly incident on the image forming lens. Moreover, 
the central portion of the image pickup range intersects 

5 tiie central portion of the illumination range of the illumi- 
nation apparatus 32B witiiin the information inputting 
space at a predetermined height from the floor surface 
(the illumination range on the floor surface illuminated 
by ttie illumination apparatus 32B is out of the image 

10 pickup range). 

[0042] As shown in Fig. 4, tiie hand pointing input 
apparatus 20 has a marking plate 40 which is disposed 
acSacent to the information inputting space. The mark- 
ing plate 40 is formed by a large number of marks 40A 

15 being recorded on a ti'ansparent flat plate in tiie form of 
matrix at an equal distance, and it is movable so as to 
cross the information inputting space along tiie direction 
which is orthogonal to the direction in which the marks 
40 A are lined up (i.e., the direction indicated by an-ow A 

20 in Fig..4). The marks 40A are colored with a color which 
can be easily recognized on tiie image (for example, a 
red color). The marking plate driving apparatus 38 
which is connected to the I/O interface 22D moves the 
marking plate 40 along the direction indicated by the 

25 an'ow A according to the instructions of the controller 
22. 

[0043] Next, as the operation of tiie present embodi- 
ment, a description of a grid point positional information 
initial setting process which is effected by the controller 
30 22 when tiie hand pointing input apparatus 20 is first set 
up will be given witii reference to tiie flowchart of Figs. 
5Atiirough5C. 

[0044] In Step 1 00. tiie marking plate 40 is moved to 
a predetermined position (which corresponds to the 

35 limit of tiie range of movement of tiie marking plate 40) 
by the marking plate driving apparatus 38. Next, in Step 
102, in the information inputting space, tiie 3-D coordi- 
nates (x. y. z) of each of tiie large number of marks 40A 
which are recorded on tiie marking plate 40 are oom- 

40 puted. In Step 104. an image of the information inputting 
space is picked up by tiie video cameras 36A and 36B 
tiirough tiie image pickup conti-ol apparatus 34. Next, in 
Step 108. tiie image of the information inputting space 
picked up by the video camera 36A (tiie image is 

45 referred to as an image A) is captured tiirough tiie 
image pickup conti^ol apparatus 34. 
[0045] In Step 108. the recognition (exti^action) of tiie 
marks 40A existing in tiie image A which is captured in 
Step 106 is effected. Next, in Step 110, the positions 
(Xa. Ya) on the image A are computed for all of tiie rec- 
ognized marks 40A. In Step 1 12, ttie 3-D coordinates (x. 
y, z) within the information inputting space and the posi- 
tions (Xa. Ya) on tiie image A are corresponded to each 
otiier for all of ttie marks 40A which exist in tiie image A, 

55 and are thereby stored in the storage device 24 as tiie 
grid point positional information from tiie video camera 
36A. 

[0046] Next, in Steps 114 to 120. similarly to the 
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abovenjescribed Steps 106 to 112, a process for the 
video, camera 36B is executed. Namely, in Step 1 14, an 
image of the information inputting space picked up by 
\ the video camera 36B (the image is refen^ed to as an 
image B) is captured through the image pickup control 5 
apparatus 34. In Step 116. the recognition (extraction) 
of the marks 40A existing in the image B which is cap- 
tured in Step 1 1 4 is effected. Next, in Step 1 1 8. the posi- 
tions (Xb, Yq) on the image B are computed for all of the 
recognized marks 40A. In Step 120. the 3-D coordi- 70 
nates (x, y. z) within the information inputting space and 
the positions (X^. Y^) on the image B are corresponded 
to each other for all of the marks 40A which exist in the 
image B. and are thereby stored in the storage device 
24 as the grid point positional information from the video 15 
camera 36B. 

[0047] Next, in Step 1 22, it is determined whether the 
marking plate 20 is moved to its terminal position (i.e., 
the position which corresponds to the end opposite to 
the predetermined portion in Step 100, within.the mov- 20 
ing range of the marking plate 40). If the determination 
is negative in Step 122, the routine proceeds to Step 
124, where after the marking plate 40 has been moved 
by the marking plate driving apparatus 38 in a predeter- 
mined direction by a fixed distance (the distance which 25 
con^esponds to the distances between the marks 40A 
on the marking plate 40), the routine returns to Step 
102. 

[0048] As described above, until the marking plate 40 
is moved to the aforementioned terminal portion, Steps so 
102 to 124 are repeated so that a large number of the 
marks 40A which are recorded on the marking plate 40 
are moved to the positions conresponding to a large 
number of grid points which are lined up in a lattice 
shape. Then, for all of the marks 40A, the 3-D coordi- 3S 
nates of each of the grkJ points within the information 
inputting space are corresponded to the positions on 
the image A. as the grid point positional information of 
the video camera 36A, and the marks are stored in the 
storage device 24. In the same way, for all of the marks 40 
40A, the 3-D coordinates of each of the grid points 
within the information inputting space are corresponded 
to the positions on the image B. as the grid point posi- 
tional information of the video camera 36B, and the 
marks are stored in the storage device 24. 45 
[0049] Further, because the marking plate 40 and the 
marking plate driving apparatus 38 are used merely for 
the aforementioned grid point positional information ini- 
tial setting process and not used for the process which 
will be described later, after the above-described proc- so 
ess has been effected, the marking plate 40 and the 
marking plate driving apparatus 38 can be removed. 
[0050] Instead of the marking plate 40 on which the 
marks 40A are recorded, the above-described process 
may be effected by using a transparent, flat marking ss 
plate on which a large number of light emitting elements 
such as LEDs or the like are disposed in the form of a 
matrix. Namely, the marking plate is moved in steps of a 



fixed distance, and the sequential switching on of the 
large number of light emitting elements is repeated at 
each of the positions in the step movement. Further, in 
order to effect the above-described process, a robot 
arm apparatus in which light emitting elements are 
mounted to a hand thereof and the hand is movable to 
any position within the information inputting space can 
be used by moving the light emitting elements to each of 
the positions conresponding to each of the grid point 
positions by the robot arm apparatus and by repeatedly 
switching on the light emitting elements. 
[0051 ] Next, with reference to the flowchart in Figs. 6A 
through 6D. a description of the pointing determination 
process will be given hereinafter. The pointing determi- 
nation process is effected regularly by the controller 22 
after the above-described grid point positional informa- 
tion initial setting process has been effected. Further, 
the pointing determination process is provided to deter- 
mine the pointing .inputted, by the information inputting 
person 10 who arrives in the information, inputting 
space. 

[0052] In Step 150, the image data representing the 
image A which is outputted from the video camera 36A 
and the image data representing the image B which is 
outputted from the video camera 36B are respectively 
captured, and on the basis of the image data from the 
captured images A and B, it is determined whether the 
information inputting person 10 has arrived in (or exists 
within) the information inputting space. 
[0053] During the pointing determination process by 
the controller 22, as shown in Figs. 10A to IOC, the illu- 
mination control apparatus 30 switches on the illumina- 
tion apparatuses 32A and 32B, alternately Accordingly, 
the image pickup control apparatus 34 controls the 
video camera 36A to pick up the image of the informa- 
tion inputfing space while the illumination apparatus 
32A is switched on, and controls the video camera 36B 
to pick up the image of the information inputting space 
while the illumination apparatus 32B is witched on. 
[0054] Since the image pickup range of the video 
camera 36A is controlled such that the illumination 
range on the floor surface illuminated by the illumination 
apparatus 32A is out of the image pickup range, even 
when there exists an object 50A which is not the subject 
to be recognized such as the baggage of the informa- 
tion inputting person 10, or rubbish or the like (see Rg. 
3) on the floor surface within the illumination range of 
the illumination apparatus 32A, the object 50A which is 
not the subject to be recognized never enters into the 
image pickup range of the video camera 36A. Further, if 
an object B which is not the subject to be recognized 
(see Fig. 3) is present on the floor surface within the 
image pickup range of the video camera 36A, since the 
object 50B which is not the subject to be recognized is 
out of the illumination range of the illumination appara- 
tus 32A, the luminance of the image portion conre- 
sponding to the object SOB which is not the subject to be 
recognized in the image A is extremely low. 
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[0055] Similarly, since the image pickup range of the 
video cangera 36B is controlled such that the illumina- 
tion range on the floor surface illuminated by the illumi- 
nafion apparatus 32B is out of the image pickup range, 
even when there exists an object 508 which is not the 
subject to be recognized such as the baggage of the 
information inputting person 10, or rubbish or the like 
(see Rg. 3) within the illumination range on the floor sur- 
face of the illumination apparatus 33. the object SOB 
which is not the subject to be recognized never enters 
into the image pickup range of the video camera 36B. 
Further, if an object 50A which is not the subject to be 
recognized is present on the floor surface within the 
image pickup range of the video camera 36B, since the 
object 50A which is not the subject to be recognized is 
out of the illumination range of the illumination appara- 
ti^ 32B, the luminance of the image portion corre- 
sponding to the object 50A which is not the subject to be 
recognized in the image B is extremely low. 
[0056] Accordingly, the determination in the above- 
described Step 150 can be effected by an extremely 
simple determination as to whether or not the image 
portion of the image A or B, for example, has a high 
luminance and has an area of a predetermined value or 
more. When the determination is negative in Step 150, 
the routine is kept in a waiting state until the determina- 
tion is affirmative. 

[0057] When the information inputting person 10 
comes into the information inputting space, the determi- 
nation in Step 150 is affirmative and the routine pro- 
ceeds to Step 152, where a reference 
point/characteristic point coordinate computation proc- 
ess is started. This process corresponds to the confut- 
ing means according to the present invention, and in the 
controller 22, it is executed concurrently with the deter- 
mination process. Hereinafter, a description of the refer- 
ence point/characteristic point coordinate computation 
process will be given with reference to the flowchart in 
Figs. 7A through 7C. 

[0058] In Step 210, data for each of the image A and 
the image B is captured from each of the video cameras 
36A and 36B. In Step 212, the image portion which cor- 
responds to the entire image of the information inputting 
person 10 is extracted from each of the captured 
images A and B. The image portion which con-esponds 
to the entire image of the information inputting person 
10 can be easily extracted by detecting the continuous 
areas of high luminance pixels larger than an area of a 
predetermined size. 

[0059] In Step 21 4, the height of the information input- 
ting person 10 is determined on the basis of the image 
portion which corresponds to the image of the entire 
body of the information inputting person 1 0. As shown in 
Fig. 11, f denotes a focal length of the image forming 
lens of the video camera 36 positioned at a point O. H 
denotes the distance between an intersection point Q of 
a vertical line passing through the point O, H denotes 
the floor surface of the information inputting space and 



the point O, R denotes the distance between the point Q 
and a point P on the floor surface on which the informa- 
tion inputting person 10 is standing, and the distance h, 
between a point P' which corresponds to the top of the 

5 head of the information inputting person 10 and the 
point P, denotes the height of the information inputting 
person 10. Further, when an angle which is formed by 
the points P, O, and Q is 6, an angle which is formed by 
the points P*, O, and Q is 0', the length of the image of 

10 the information inputting person 10 which is formed on 
the receptor surface of the area sensor of the video 
camera 36 is h*. a point p denotes an image forming 
point on tiie receptor surface which corresponds to tiie 
point P, a point p' denotes an image forming point on the 

15 receptor surface which con-esponds to the point P', r 
denotes the distance between the center o of the recep- 
tor surface and the point p'. and r' denotes tiie distance 
between tiie center o of tiie receptor surtace and tiie 
point p', tiie angles 8, 9' and tiie distances r. r' are deter- 

20 mined from tiie following equations (1) to (4); 

G=tan'\R/H) (1) 

e'=tan'''{FV(H-h)} (2) 

25 

r=fe (3) 

r*=fO' (4). 

30 [0060] Therefore, the height h of ttie information Input- 
ting person 1 0 and the distance R can be determined by 
tiie following equations (5) and (6); 

h =H{1-tan(r/0/lan(r/0} (5) 

35 

R= Htan(r/ff) (6) 

[0061 ] Because tiie distance H and the focal lengtii f 
are already known, in Step 214, tiie distances r and r' 

40 are determined from one of the image A and the image 
B picked up by tiie video cameras 36A and 36B. and 
these determined distances r and r* are substituted in 
tiie equation (5), whereby tiie height h of tiie information 
inputting person 10 can be determined. Further, in Step 

45 214, tiie distance r is determined from each of tiie 
images A and B, and each distance r is then substituted 
in tiie equation (6) so that each of the distances R is 
determined, whereby the position (2-D coordinates) of 
tiie information inputting person 10 on the floor surface 

50 can be determined. 

[0062] Next, in Step 216, tiie 3-D coordinates (xq, yo, 
zq) of a reference point Pq of the information inputting 
person 10 is determined on tiie basis of tiie height h of 
tiie information inputting person 10. and tiie position of 

55 the information inputting person 10 on the floor surface 
as determined in Step 214. Moreover, the point (i.e.. the 
point Pq which is shown in Fig. 17) which corresponds 
to. for example, the back of tiie information inputting 
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person 10 can be used for the reference point Pq. In this 
case, jthe height (e.g., the value zq) between the refer- 
ence point Pq which corresponds to, for example, the 
* back of the information inputting person 10 and the floor 
surface is calculated on the basis of the height h of the 5 
information inputting person 10 and the position of the 
Information inputting person 10 on the floor (plane coor- 
dinates) is set as the plane coordinates (for example, 
the values of xq and yo), whereby the 3-0 coordinates 
for the reference point Pq can be determined. Instead of 10 
the point which conesponds to the back of the informa- 
tion inputting person 10. a point corresponding to the 
chest of the information inputting person 10, a point cor- 
responding to the shoulder joint of the information input- 
ting person 1 0. or the like can be used. 15 
[0063] In Step 21 8. on the basis of the configuration of 
the image portion which corresponds to the whole body 
image of the information inputting person 10 on the 
image A and the image B. it is determined whether the 
information inputting person 1 0 is pointing (i.e.. perform- 20 
ing a pointing action) to the display 12 with his or her fin- 
ger. Because the direction of the display 12 as seen by 
the information inputting person 10 is already known, 
the determination in St^ 218 can be accomplished by, 
for example, determining whether or not the portion pro- 25 
jecting toward the display 12. as seen from the informa- 
tion inputting person 10 is present at the height 
determinable as the position of the hand of the informa- 
tion inputting person 10. in the image portion corre- 
sponding to the full-length image of the information 30 
inputting person 10. 

[0064] Therefore, from a state in which the person 1 0 
is standing upright (Fig. 12A). when the information 
inputting person 10 lifts his or her hand and directs the 
hand toward the display 12. it can be determined that 35 
the information inputting person 10 is canrying out a 
pointing action. When the determination is negative in 
Step 218. the 3-D coordinates for the characteristic 
point (which will be later described in more detail) are 
not computed and the routine returns to Step 210. 40 
Steps 210 to 218 are repeated until the information 
inputting person 10 carries out a pointing action. 
[0065] When the information inputting person 10 car- 
ries out the pointing action, the determination in Step 
21 8 is affirmative, and the routine proceeds to Step 220. 45 
In Step 220. on the basis of the image data representing 
the captured image A from the video camera 36A. the 
characteristic point Fx of the information inputting per- 
son 10 within the image A is extracted, and the positions 
(Xyv. Ya) of the characteristic point Fx on the image A so 
are thereby computed. Points or the like which corre- 
spond to the tip of the finger which is pointing toward the 
display 1 2 can be used for the characteristic point Fx of 
the information inputting person 10. In this case, within 
the image portion representing the image of the entire ss 
body of the information inputting person 10, the position 
of the tip of the portion projecting toward the display 12 
from the height position which can be determined to 



correspond to the hand of the Information inputting per- 
son 10 can be corriputed as the position of the charac- 
teristic point Fx- 

[0066] Accordingly, when the image of the hand of the 

information inputting person 1 0 is picked up by the video 
camera 36A as shown in Fig. 13A, the coordinates (Xa. 
Ya) of the characteristic point Fx are computed as the 
position of the characteristic point Fx- 
[0067] In Step 222. on the basis of the grid point posi- 
tional Information of the video camera 36A which Is 
stored in the storage device 24, alt of the grid points 
whose positions on the image A are within the range of 
(XA±dX, YA±dY) (see the range circled by a section line 
in Fig. 13B) are detected. Further, the magnitude of 
each of dX and dY are determined on the basis of the 
distances between grid points (i.e., the distances 
between the marks 40A) such that one or more of the 
grid points is extracted. 

[0068] Further, in the present embodiment, a wide 
angle lens is used for an. Image forming lens of the video 
cameras, and provkied that dX and dY are fixed values, 
a large number of grid points are contained within tfie 
range of (XAidX. YA±dY). which leads to a decrease in 
the accuracy of computing the 3-D coordinates of the 
characteristic point Fx which will be described later. For 
this reason, dX and dY are set such that the further the 
distance from dX and dY to the video camera on the 3- 
D coordinates, the smaller the values. Accordingly, the 
range which is equal to (XA±dX. YA±dY) on the 3-D 
coordinates forms a conical shape (or a conical ellipse 
shape) whose bottom surface is disposed on the side of 
the video camera. 

[0069] In Step 224, in the same manner as the afore- 
mentioned Step 220, on the basis of the image data 
which represents the image B which is captured from 
the vkJeo camera 36B. the characteristic point Fx of the 
information inputting person 10, within the image B is 
extracted, and the position (X^. Yq) of the characteristic 
point Fx on the image B is computed. In Step 226, in the 
same manner as the aforementioned Step 222, all of 
the grid points within the range of (XB±dX, YerfY) are 
detected on the basis of the grid point positional infor- 
mation of the video camera 36B which is stored in the 
storage device 24. 

[0070] Next, in Step 228, the common grid points 
which are extracted from the image A and the image B 
are determined. Accordingly, only the plurality of the 
grid points which are disposed adjacent to the charac- 
teristic point Fx within the information inputting space 
are extracted. In Step 230, the 3-D coordinates of the 
common grid points extracted from the image A and the 
image B are captured from the grid point positional 
information. 

[0071] In the present embodiment, as will be 
described later, the 3-D coordinates of the characteristic 
point Fx are computed through interpolation from the 3- 
D coordinates of the plurality of grid points which exist 
adjacent to the characteristic point Fx in the infbrrration 
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inputting space (in more detail, the coordinate values of 
the 3-D c9ordinates of the characteristic point are deter- 
mined by the weighted mean value of the coordinate 
values of 3-D coordinates of the plurality of grid points). 
For this reason, previous to the computation of the 3-D 
coordinates of the characteristic point Px, in the follow- 
ing Step 232, on the basis of the positions on the image 
A and the image B of each of the common grid points 
which are extracted from the image A and the image B. 
the position (Xa. Y/O of ttie characteristic point Px on the 
image A. and the position (Xb. Yb) of the characteristic 
point Px on the image B. the interpolation ratio (the 
weighting for the coordinate values of the 3-D coordi- 
nates of each of the grid points) is determined from the 
3-D coordinates of each of the common grid points 
which are extracted from the image A and the image B. 
For example, the interpolation ratio can be determined 
such that the weighting for the coordinate values of the 
3-D coordinates of each of the grid points which are dis- 
posed adjacent to the characteristic point on the image 
A and the image B may be increased. 
[0072] In Step 234, the 3-D coordinates (Xx, Yx, Zx) 
of the characteristic point Px are computed on the basis 
of 3-D coordinates of the common grid points which are 
extracted from the image A and the image B, and the 
interpolation ratio which is determined in the aforemen- 
tioned Step 232. 

[0073] As described above, the 3-D coordinates (Xx, 
Yx. Zx) of the characteristic point Px are computed, and 
thereafter, the routine returns to Step 210, where the 
processes from Step 210 are repeated. In this way, in 
the reference point/characteristic point coordinates 
computation process, because the reference point Pq 
and the characteristic point Px of the information input- 
ting person 10 (provided that the information inputting 
person 10 is can-ying out a pointing action) are com- 
puted repeatedly, tiie values of 3-D coordinates of tiie 
reference point Pq and the characteristic point Px which 
are computed by the reference point/characteristic point 
coordinate computation process are sequentially 
renewed in response to a change in the posture or the 
nmtion of the information inputting person 10. 
[0074] The controller 22 starts the reference 
point/characteristic point coordinate computation proc- 
ess in Step 152 of the pointing determination process 
(Figs. 6 A through 6D), and ttiereafter, executes the 
processes downward from Step 154 in tiie pointing 
determination process concunrentiy with tiie aforemen- 
tioned reference point/characteristic point coordinate 
computation process. Namely, in Step 154, tiie control- 
ler 22 executes a distance conversion conditions setting 
process. A description of the distance conversion condi- 
tions setting process will be given with reference to the 
flowchart of Figs. 8A tiirough 8C. Furtiier. the distance 
conversion conditions setting process corresponds to 
the conversion conditions setting means of the tiiird 
aspect of tiie present invention. 
[0075] In Step 250, in a virtual 3-D space represented 



by a tiiree dimensional image displayed on tiie display 
12, tiie 3-D coordinates at a position which corresponds 
to a substantially intermediate position (i e., tiie neutral 
point) of the virtual 3-D space in tiie depth direction 

5 tiiereof. as seen from the information inputting person 
10, is determined on tiie basis of the coordinate data or 
tiie like which is stored in tiie storage device 24 (which 
may also be computed by a functional equation), and 
the position of tiie neutral point on the display screen of 

10 tiie display 12 (display position of the neutral point) is 
computed, and the information processing apparatus 14 
is instructed to display a marker on the display position 
which has been computed on the displaying screen of 
the display 12. 

IS [0076] Therefore, the information processing appara- 
tus 14 displays the marker at tiie display position on the 
display screen of tiie display 12, the displayed marker 
may be visible to tiie information inputting person 10 as 
if it were positioned at the neutral point of tiie virtual.3-D 

20 space. Moreover, a circular cursor 52 which is shown in 
Fig. 18 or the like can be used for the marker. However, 
any configuration of a marker can be employed if it sat- 
isfies the conditions under which the information input- 
ting person 10 can easily recognize the configuration. 

25 [0077] Next, in Step 252, tiie information processing 
apparatus 14 is instructed to display on the display 12, 
a message which requires the information inputting per- 
son 10 to can^y out a pointing action at tiie marker dis- 
played on tiie display 12. Accordingly, the information 

30 processing apparatus 14 displays the message on tiie 
display 12, and the information inputting person 10 in a 
state in which the person 10 is standing upright (Fig. 
12A). lifts his hand and directs the hand toward tfie dis- 
play 12, and in ttie direction of the marker in tiie virtual 

35 3-D space represented by a tiiree dimensional image 
displayed on tiie display 12, and bends or stretches his 
or her arm in accordance with tiie distance to tiie 
marker in the virtual 3-D space. 
[0078] When the infornnation inputting person 10 

40 points to an object which has been recognized by the 
information inputting person 1 0 to be located at a distant 
position in the virtual 3-D space, the person 10 
stretches out his or her arm (see Fig. 14A). When tiie 
information inputting person 10 points to an object 

45 which has been recognized by the information inputting 
person 10 to be located adjacent to him or her, the per- 
son 10 bends his or her arm (see Fig. 14C). As 
described above, because tiie marker can be seen as if 
it were positioned at a neutral point in the virtual 3-D 

so space, in the action of pointing to ttie marker, tiie infor- 
mation inputting person 10's arm is in a half-bent state 
as shown in Rg. 14B. 

[0079] In Step 254, the results (3-D coordinates of the 
reference point Pq and the characteristic point Px) of tiie 
55 reference point/characteristic point coordinate compu- 
tation process are captured, and in Step 256, it is deter- 
mined whetiier the information inputting person 10 
carries out tiie pointing action by pointing to the display 
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screen of the display 12. In the reference point/charac- 
teristiq point coordinate computation process, since the 
3-D coordinates of the characteristic point Px are com- 
.puted only when the information inputting person 10 
carries out the pointing action (i.e., when the determina- 
tion in Step 21 8 is affirmative), the determination in Step 
256 is effected by determining whether 3-D coordinates 
of the characteristic point Px are computed. 
[0080] When the determination is negative in Step 
256, the routine returns to Step 254, and Step 254 and 
Step 256 are repeated until the information inputting 
person 10 cames out the pointing action. When the 
determination in Step 256 is affirmative, the routine pro- 
ceeds to Step 258, where the distance k between the 
reference point Pq and the characteristic point Px is 
computed from the 3-D coordinates of the reference 
point Pq and the characteristic point Px which are cap- 
tured in Step 254. The distance 1^ is corresponded to the 
distance L^jd (virtual distance) between the infornetion 
inputting person 10 and the neutral point in the virtual 3- 
D space containing the information inputting person 10, 
and is stored in the storage device as a distance kmid- 
[0081 ] Next, in Step 260, the distance conversion con- 
ditions setting process instructs the information 
processing apparatus 14 to display on the display 12, a 
message which requires the information inputting per- 
son 10 to can'y out an arm bending or stretching action 
over the complete range of frie stroke of the arm. 
Accordingly, the information processing apparatus 14 
displays the message on the display 12. The informa- 
tion inputting person 10 repeats the stretching motion in 
which the information inputting person 10 stretches an 
arm out straight and the bending action In which the 
information inputting person 10 bends an arm. 
[0082] Next, in Step 262, the distance conversion con- 
ditions setting process captures the process results of 
the reference point/characteristic point coordinate com- 
putation process. In Step 264, it is determined whether 
the information inputting person 10 carries out the 
action of pointing to the display saeen of the display 12. 
When the determination is negative in Step 264, the 
routine returns to Step 262. Step 264 and Step 262 are 
repeated until the information inputting person 10 car- 
ries out the pointing action. When tiie determination in 
Step 264 is affirmative, tiie routine proceeds to Step 
266. 

[0083] In Step 266, ttie distance k between tiie refer- 
ence point Pq and tiie characteristic point Px is com- 
puted on the basis of the 3-D coordinates of the 
reference point Pq and tiie characteristic point Px which 
are captured in Step 262, and is stored and it is deter- 
mined whether any change in the distance k exists and 
whettier the direction of tiie change in the distance k 
changes from an increasing direction to a decreasing 
direction, or from a decreasing direction to an increas- 
ing direction. The determination will be unconditionally 
that no change exists when Step 266 is executed for the 
first time and the second time. However, when Step 266 



is executed for the second time and tiiereafter, the value 
of the distance k which is computed this time is com- 
pared to the values of the distance k which has been 
computed for tiie previous times, and tiie changing 

5 direction of the distance k is determined. When Step 
266 is executed for the third time and tiiereafter, tiie 
aforementioned determination is carried out by compar- 
ing ttie direction of tiie change in the distance k deter- 
mined during tiie curent determination witii tiie 

10 direction of tiie change in tiie distance k determined 
during tiie previous determination. 
[0084] When the aforementioned determination is tiiat 
no change exists, the routine returns to Step 262, and 
Steps 262 to 266 are repeated. On the other hand, 

15 when tiiere is a change in tiie direction of the distance 
k, the information inputting person 10*s arm may be 
substantially in a stretched out straight state (see Fig. 
14A) or in a bent state (see Fig. 14C). Accordingly, tiie 
determination in Step 266 is affirmative, and the routine 

20 proceeds to Step 268, where tiie distance k between 
tiie reference point Pq and the diaracteristic point Px at 
this time (the distance k shows a maximum value in a 
state in which tiie information inputting person 10's arm 
is sb'etched out straight, and a minimum value in a state 

25 in which tiie information inputting person 10's arm is 
bent) is stored. Next, in Step 270. it is determined 
whetiier botti tiie maximum value k„ax the mini- 
mum value kmin are obtained. If tiie determination is 
negative in Step 270. the routine goes back to Step 262, 

30 and Steps 262 to 270 are repeated. 

[0085] After botii tiie maximum value k^ax 
minimum value k^jn are obtained in Step 270, the rou- 
tine proceeds to Step 272, where the distance conver- 
sion conditions are set for converting the distance k 

35 between the reference point Pq and tiie characteristic 
point Px of ttie information inputting person 10 into ttie 
distance L (virtual distance) between the information 
inputting person 10 and the position which is pointed to 
by tiie information inputting person 10. These distance 

40 conversion conditions can be determined by using tiie 
least square metiiod such tiiat, as shown in Fig. 15. for 
example, tiie distance k^in is converted into tiie dis- 
tance Lmin. ttie distance IVnid 'S converted into the dis- 
tance L^id, and the distance k^ax converted into tiie 

45 distance L^ax> wherein tiie distance (virtual distance) 
between tiie front most position as seen from the infor- 
mation inputting person 10 and ttie information inputting 
person 10 is L^jn, tiie distance between the deepest 
position as seen from tiie information inputting person 

so 10 and ttie information inputting person 10 is L^ax* ^ 
tiie distance between ttie neutral point and the informa- 
tion inputting person 10 is L^id- 
[0086] The length or the like of the arm may be differ- 
ent depending on the individual information inputting 

55 person 10, and accordingly, the widtti of ttie change 
(^ax'^min) distance k when ttie information input- 
ting person 10's arm is stiretched/bent may also differ 
depending on the individual. However, the above- 
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described distance conversion conditions setting proc- 
ess is carried out eacli time an information inputting per- 
son 10 comes into the information inputting space. The 
process determines the maximum value and the mini- 
mum value of the distance k between the reference 5 
point Pq and the characteristic point Px. and sets the 
distance conversion conditions so that the width of the 
change (kfnax'^in) distance k corresponds to the 
width of the change (Ln,ax-Lmin) distance L 

between each position within the virtual 3-D space and 10 
the information inputting person 10. As a result, when 
another information inputting person 10 comes into the 
information inputting space, the appropriate conversion 
conditions can be provided as distance conversion con- 
ditions in accordance with the length or the like of the 15 
arm of the newly-arrived information inputting person 
10. 

[0087] Further, in the above-described distance con- 
version conditions, because the conversion characteris- 
tics are fixed in view of the distance k between the 20 
reference point Pq and the characteristic point Px when 
the information inputting person 10 carries out the 
action of pointing to a neutral point, the conversion char- 
acteristics can be corresponded to the senses of the 
information inputting person 10 so that the distance L 25 
between the position which is pointed to by the informa- 
tion inputting person 10 and the information inputting 
person 10 can be determined accurately. 
[0088] The distance conversion conditions setting 
process is candied out as described above, and thereaf- 30 
ter. in Step 156 (see Fig. 6), a click action speed setting 
process is executed. A description of the dick action 
speed setting process will be given with respect to the 
flowchart in Fig. 9. Further, the click action speed set- 
ting process con^esponds to the threshold value setting 35 
means according to the seventh aspect of the present 
invention. 

[0089] In Step 290. the click action speed setting proc- 
ess instructs the information processing apparatus 14 to 
display on the display 12. a message which requests 40 
the information inputting person 10 to can-y out a click 
action. The information processing apparatus 14 
causes the display 12 to display the message. The 
present embodiment fixes the click action as a motion in 
which the information inputting person 10 moves his 4S 
hand forward quickly (see Fig. 16A. which is refen'ed to 
as a forward dick) and a motion in which the information 
irputting person 10 moves his hand backward quickly 
(see Fig. 16B. which is referred to as a backward dick). 
When tiie aforementioned message is displayed on the so 
display 12. the information inputting person 10 bends or 
stretches out an arm, and repeats the fonvard click 
action and tiie backward click action. 
[0090] In Step 292. the process results from the refer- 
ence point/characteristic point coordinate conputation 55 
process (tiie 3-D coordinates of tiie reference point Pq 
and tiie characteristic point Px) are captured. In Step 
294, it is determined whettier the information inputting 



person 10 is carrying out the action of pointing to ttie 
display 12. When the determination is negative in Step 
294. the routine returns to Step 292, and Step 292 and 
Step 294 are repeated until ttie information inputting 
person 10 carries out the pointing action. When tiie 
determination in Step 294 is affirmative, ttie routine pro- 
ceeds to Step 296. 

[0091 ] In Step 296. the distance k between the refer- 
ence point Pq and the characteristic point Px is com- 
puted from ttie 3-D coordinates of the reference point Pq 
and ttie characteristic point Px. which are captured in 
Step 292. Further, Step 296 is carried out repeatedly 
However, when Step 296 is carried out for the second 
time and thereafter, tiie speed V of ttie change in ttie 
distance k (the speed at which tiie position of the char- 
acteristic point Px moves to the position of ttie reference 
point Pq) is computed on ttie basis of the difference of 
ttie value of ttie distance k which has been computed 
ttiis time and ttie value of tiie distance k which was com- 
puted for tiie previous time, and tiie computing results 
are stored. 

[0092] In Step 298. tiie message requesting "carry out 
dick action" is displayed on ttie display 12 and tiiereaf- 
ter, it is determined whether a predetermined time has 
been passed. When ttie determination is negative, tiie 
routine returns to Step 292. and Steps 292 to 298 are 
repeated. Accordingly, from the time when tiie message 
requesting "carry out click action" until the time when a 
predetermined time has passed, the speed V of the 
change in the distance k between the characteristic 
point Px and tiie reference point Pq) is computed and 
stored repeatedly 

[0093] When the determination is affirmative in Step 
298, the routine proceeds to Step 300. In Step 300. tiie 
previously conrputed and recorded speed of the change 
V is captured and ttie speed of the click action Vq (which 
corresponds to a tiireshold value according to the f iftti to 
seventti aspects of tiie present invention) is set and 
stored based on ttie progress of the speed of ttie 
change V while tiie information inputting person 10 is 
performing tiie click action once. The click action speed 
Vq is used as a tiireshold value for determining whether 
tiie information inputting person 10 has carried out ttie 
dick action in a process which will be described later. 
For ttiis reason, in order to reliably determine whether 
ttie information inputting person 10 canried out tiie click 
action, for example, a value which is slightiy less tiian 
ttie mean value of the speed of tiie change V in a single 
dick action by the information inputting person 10 can 
be set for tiie click action speed Vq. In addition, ttie min- 
imum value of the speed of the change in a single click 
action by the information inputting person 10 can be set 
for ttie click action speed Vq. 
[0094] The transition speed (speed of ttie change V) 
of the characteristic point Px when the information 
inputting person 10 carries out ttie click action by bend- 
ing or sti^etching an arm may vary depending on tiie 
individual information inputting person 10. However. 
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since the above^iescribed click action speed setting 
process is carried out each time an infornnation inputting 
person 10 comes into the information inputting space. 
*when a new information inputting person 10 comes into 
the information inputting space, an appropriate value in 5 
accordance with the physique or the muscular strength 
of the new information inputting person 10 who comes 
into the information inputting space is newly set as the 
click action speed Vq. 

[0095] After the above<lescribed click action speed 10 
setting process has been completed, the routine pro- 
ceeds to Step 158 in the pointing determination process 
(see Figs. 6A through 6D). In and after Step 158, the 
determination process for determining whether the 
information inputting person 10 can'ies out the pointing 75 
action is executed. Namely, in Step 158. the process 
results by the reference point/characteristic point coor- 
dinate computation process (the 3-D coordinates of the 
reference point Pq and the characteristic point Px) are 
captured, in Step 160. it is determined whether the 20 
information inputting person 10 carries out a pointing 
action. 

[0096] When the determination is negative in Step 
160, the routine proceeds to Step 194, wherein it is 
determined whether the information inputting person 10 25 
has left the information inputting space. In the same 
manner as Step 150. an extremely sinrple determination 
is made in which it is determined whether the image 
portion whose intensity is high and which has an area 
which is greater than or equal to a predetermined value 30 
is lost from the image A and the image B. When this 
determination is negative, the routine returns to Step 
158, and Steps 158, 160, and 194 are repeated. 
[0097] Further, when the determination is affirmative 
in Step 160, the routine proceeds to Step 162. where, 35 
on the basis of the 3-D coordinates of the reference 
point Pq and the characteristic point Px which are cap- 
tured in the previous Step 158. the coordinates (plane 
coordinates) of an intersection point (see the point S in 
Figs. 17A and 17B) of a virtual line connecting the 40 
above-described reference point and the characteristic 
point (see virtual line 54 in Rgs. 17A and 17B). and a 
plane including the display screen of the display 12 are 
computed. In Step 164. on the basis of the coordinates 
of the intersection point S which is obtained in Step 1 62, 45 
it is determined whether the information inputting per- 
son 1 0 is pointing at the display screen of tiie display 12. 
When this determination is negative, the routine pro- 
ceeds to Step 194. and Steps 158 to 164. and Step 194 
are repeated. so 
[0098] Meanwhile, when the determination is affirma- 
tive in Step 164, it can be determined that the informa- 
tion inputting person 10 is pointing at tiie specific 
position within the virtual 3-D space represented by a 
three-dimensional image displayed on the display 12. ss 
For this reason, the routine proceeds to Step 166, 
where, on the basis of the 3-D coordinates of ttie refer- 
ence point Pq and the characteristic point Px which are 



captured in advance, as the direction in which tiie posi- 
tion pointed to by the information inputting person 10 
exists, tiie direction in which ttie virtual line connecting 
tiie reference point Pq and the characteristic point Px 
extends (i.e.. the direction from tiie reference point Pq to 
tiie characteristic point Px) is computed. 
[0099] In Step 168. on tiie basis of the 3-D coordi- 
nates of tiie reference point Pq and the characteristic 
point Px which are captured previously, the distance k 
between the reference point Pq and tiie characteristic 
point Px is computed. Then, in accordance with ttie dis- 
tance conversion conditions (see Fig. 15) which are set 
in the above-described distance conversion conditions 
setting process (see Figs. 8A through 8C), as shown in 
Fig. 18, the distance k between the reference point Pq 
and the characteristic point Px is converted into tiie dis- 
tance L (virtual distance) between the information input- 
ting person 10 and the position pointed to by the 
information inputting person 10. In Step 170, the posi- 
tion pointed to by the information inputting person 10 is 
determined to be the position which is spaced apart 
from ttie reference point Pq by the distance L which has 
been obtained in Step 168 and which is in the direction 
computed in Step 1 66 so tiiat the 3-D coordinates of the 
position in tiie virtual 3-D space, pointed to by the infor- 
mation inputting person 10 are computed. 
[01 00] Accordingly, even when the information input- 
ting person 10 points to an arbitrary position within tiie 
virtual 3-D space represented by a three dimensional 
image displayed on the display 12. it is possible to 
determine tiie 3-D coordinates of the position which is 
pointed to by the information inputting person 10. Fur- 
ther, the above-described Steps 166 to 170 correspond 
to the determining means of tiie present invention. 
[01 01 ] In Step 1 72, ttie 3-D coordinates of tiie position 
pointed to by tiie information inputting person 10 which 
has been computed in Step 170 are converted into ttie 
position (2-D coordinates) which is pointed to on the dis- 
play screen of tiie display 12 (tiie 2-D coordinates can 
be obtained tiirough computation using a functional 
equation) in accordance witii the coordinate data which 
is stored in ttie storage device 24, and ttiis process 
instructs the information processing apparatus 14 to 
display tiie cursor 52 at the aforementioned position 
(see Fig 18). The size of the cursor 52 displayed on the 
display 12 can be decreased as tiie length L which has 
been determined in Step 168 increases. 
[0102] In tiie present embodiment, ttie information 
inputting person 10 can instruct ttiat ttie magnitude 
(zoom-in or zoom-out) of a tiiree dimensional image 
which is displayed on the display 12 be changed. 
Namely, in order to distinguish the motion in which the 
information inputting person 10 is pointing to a specific 
position within ttie virtual 3-D space from the motion in 
which the information inputting person 10 insti'ucts ttiat 
the display magnification of a ttiree dimensional image 
which is displayed on ttie display 12 be changed, for 
exanple, when the information inputting person 10 car- 
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ries out a pointing action, it is previously determined 
that the ^information inputting person 10*s hand is 
formed in a shape in which the thumb and all the fingers 
except for the forefinger are bent (see Fig. 19A), and 
when the information inputting person 10 instructs the 
display magnification to be changed, the information 
inputting person 10*8 hand is in a specific shape in 
which all the fingers except for the thumb are stretched 
out (see Fig. 19B). 

[0103] In Step 1 74, it is determined whether the infor- 
mation inputting person lO's hand is in a specific shape 
(the shape in which all the fingers except for the thumb 
are stretched out). Namely, it is determined whether the 
information inputting person 1 0 instructs that the display 
magnification of a three dimensional image be changed. 
Further, this Step 174 con^esponds to the determining 
means according to the fourth aspect of the present 
invention. 

[0104] When the determination is negative in Step 
174, it is determined that the information inputting per- 
son 10 is not candying out the motion to instruct the dis- 
play magnification of a three dimensional image to be 
changed, and the routine proceeds to Step 176. 
wherein it is determined whether the distance k which 
has been computed previously in Step 168 is changed. 
Step 168 is earned out repeatedly while the information 
inputting person 10 is performing the pointing action of 
pointing to the display screen of the display 12 (while 
the determination is affirmative in Step 164). However, 
when tiie distance k is computed for the first time in 
Step 168, because it cannot be determined whether the 
distance k has changed, the determination is uncondi- 
tionally negative in Step 176. 
[0105] When the determination is affirmative in Step 
176, ttie routine proceeds to Step 178, wherein the 
speed of tiie change V in tiie distance k is computed, 
and it is thereby determined whettier the computed 
speed of the change V is greater tfian or equal to a 
threshold value (i.e.. tiie click action speed Vq which is 
set by tiie click action speed setting process). Moreover, 
also in Step 178, when the distance k is computed for 
the first time in Step 168, because the speed of tiie 
change V in tiie distance k cannot be computed, the 
determination is unconditionally negative. When the 
determination is negative in Step 176 or in Step 178. it 
is tiiereby determined that the information inputting per- 
son 10 is not candying out tiie dick action, and the rou- 
tine proceeds to Step 194. 

[0106] Moreover, when the determination is affirma- 
tive, respectively in Step 176 and Step 178, it is thereby 
determined that tiie information inputting person 10 has 
caried out tiie click action, and in Step 180, tiie direc- 
tion of the change in the distance k is determined, and 
the routine is branched in accordance with tiie results of 
the determination. When tiie distance k changes in the 
direction of inaease, it can be determined that the infor- 
mation inputting person 10 has can-ied out a forward 
click action in which tiie information inputting person 10 



moves his or her hand forward quickly The routine pro- 
ceeds to Step 182, wherein information indicating that 
the fonward click action has been carried out is output- 
ted to the information processing apparatus 14, and the 

5 routine proceeds to Step 194. Further, when tiie dis- 
tance k changes in tiie direction of decrease, it can be 
determined ttiat the information inputting person 10 has 
carried out a backward click action in which tiie informa- 
tion inputting person 1 0 bends his or her hand backward 

10 quickly. Therefore, the routine proceeds to Step 184, 
wherein information indicating tiiat the backward click 
action has been carried out is outputted to the informa- 
tion processing apparatus 14, and the routine proceeds 
to Step 194. 

15 [01 07] When information indicating tiiat "forward click 
action" or "backward click action" has been carried out 
is outputted to tiie information processing apparatus 14, 
it is determined that the current position pointed to by 
tiie information inputting person 10 is clicked by tiie 

20 information inputting person 1 0. When the "fonA/ard click 
action" is carried out, tiie first process corresponding to 
the current position pointed to by tiie. information input- 
ting person 10 is executed. When the "backward click 
action" is carried out, tiie second process correspond- 

25 ing to the cun-ent position pointed to by tiie information 
inputting person 10 (tiiis process is different from tiie 
first process) is executed. Further, the first and second 
processes correspond to the predetermined process 
according to tire fifth aspect of the present invention (in 

30 more detail, tiie first predetermined process and tiie 
second predetermined process according to the sixtii 
aspect of tiie present invention), and. for example, a 
process relating to a position pointed to by tiie informa- 
tion inputting person 10 witiiin a virtual 3-D space (a 

35 process relating to an object which is located at tiie 
position pointed to by tiie information inputting person 
10) (a description of an example thereof will be 
described later). 

[pi 08] In this way. Steps 1 78 to 1 84 correspond to tiie 

40 information processing apparatus 14 which actually 
executes tiie first process and tiie second process, and 
tiie processing means according to the f iftii aspect of 
tiie present invention (in more detail, the sixtii aspect of 
tiie present invention). 

45 [0109] When the determination is affirmative in Step 
1 74, tiie routine proceeds to Step 186, where it is deter- 
mined whetiier tiie distance k which Is computed in the 
previous Step 168 has changed. Furtiier, also in Step 
186, when tiie distance k is computed in Step 168 for 

so tiie first time, because it cannot be determined whether 
tiie distance k has changed, the determination is uncon- 
ditionally negative. When tiie determination is negative 
in Step 186. although tiie information inputting person 
10's hand is shaped so as to insf uct that the display 

55 magnification of a three dimensional image be changed 
in view of the fact that it is not instructed as to what 
degree tiie display magnification needs to be changed, 
tiie routine proceeds to Step 194. 
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[01 1 0] When the determination is affirmative in Step 
186. the routine proceeds to Step 188. where the 
amount of the change in the distance k is computed, the 
'amount of the change in the distance k is converted into 
the amount of the change of the display magnification of 
a three dimensional image. In Step 190. the display 
magnification of a three dimensional image to be dis- 
played on the display 12 is changed in accordance with 
the amount of the change of the display magnification 
which is conputed in Step 188. And the information 
processing apparatus 14 is instructed to again display a 
three dimensional image on the display 12 so that the 
cun'ent position pointed to by the information inputting 
person 10 (i.e., the position which corresponds to the 3- 
D coordinates computed in Step 170) is located sub- 
stantially at the center of the display screen of the dis- 
play 12. 

[0111] Accordingly, for example, as shown in Fig. 20A, 
in a state in which a three dimensional image represent- 
ing a part of the universe displayed on the display 12, 
and the information inputting person 10 is pointing at a 
space shuttle 54 in the three dimensional image, when 
an enlarged display (zoom-in) of a three dimensional 
image is instructed by the information inputting person 
10, as shown in Fig. 20B. the space shuttle 54 is located 
at the center of the display screen of the display 12, and 
a three dimensional image is displayed again in such a 
way that an enlarged three dimensional image is dis- 
played in accordance with the amount of the change of 
the display magnification. 

[01 1 2] Further, when a display of a three dimensional 
image is instructed to be reduced (zoom-out) by the 
information inputting person 10 in the same state as 
described above, as shown in Fig. 20C. the space shut- 
tle 54 is located at the center of the display screen of the 
display 12, and a three dimensional image is displayed 
again in such a way that a reduced three dimensional 
image is displayed in accordance with the amount of the 
change of the display magnification. 
[01 1 3] In this way. Steps 1 86 to 1 90 correspond to the 
information processing apparatus 14 which actually 
controls the display of the 3-D image, and the display 
control means according to the fourth aspect of the 
present invention. Because the virtual distance between 
each of the positions pointed to by the information input- 
ting person 10 within a virtual 3-D space and the infor- 
mation inputting person 10 changes in response to the 
change in the aforementioned display magnificat'on, in 
the aforementioned Step 190, the distance conversion 
conditions are corrected in accordance with the change 
in the virtual distance. Next, in Step 192. the cursor 152 
is displayed again in accordance with the three dimen- 
sional image which is displayed again, and the routine 
proceeds to Step 194. 

[0114] When the information inputting person 10 who 
comes into the information inputting space carries out a 
pointing action at an arbitrary portion within a virtual 3- 
D space, which is represented by a three dimensional 



image displayed on the display 12, the cursor 52 is dis- 
played at a position which corresponds to the position 
pointed to by the information inputting person 10 on the 
screen of the display 12. When the information inputting 

5 person 10 carries out a lonward click action" or "back- 
ward click action", a desired process (e.g., a process 
related to the position pointed to by the information 
inputting person 10) can be executed. 
[0115] When the information inputting person 10's 

10 hand is in a shape in which all the fingers except for the 
thumb are stretched out, the arm of the information 
inputting person 10 is bent or stretched out so that the 
display magnification of the three dimensional image 
which is displayed on the display 1 2 has a desired mag- 

75 nification. the three dimensional inr^ge can obtain the 
desired magnification and can be displayed again 
around the position which is pointed to by the informa- 
tion inputting person 10. If. however, the extent of tiie 
display magnification of the tiiree dimensional image is 

20 insufficient even when the arm of tiie information input- 
ting person 10 has been bent or stretched out to its full- 
est extent, tiien if the shape of the hand is changed so 
tiiat tiie ttiumb and all tiie fingers except for the forefin- 
ger are closed into a fist and the arm is returned to its 

25 original bent or stretched out position, by tiien repeating 
tiie magnification instruction action, the magnification of 
tiie display may be increased or reduced even further. 
[0116] When the information inputting person 10 
leaves tiie information inputting space, the determina- 

30 tion is affirmative in Step 194, tiie execution of the refer- 
ence point/characteristic point coordinate computation 
process (Figs. 7A ttirough 7C) is terminated in Step 
196, and thereafter, tiie routine returns to Step 150, and 
waits until the information inputting person 10 comes 

35 again into the information inputting space. When the 
information inputting person 10 comes again into the 
information inputting space, the processes in and after 
Step 152 are repeated. 

[0117] In the confroller 22 of ttie hand pointing input 
40 apparatus 20. tiie position on tiie display screen of tiie 
display 12 pointed to by tiie information inputting person 
10 and tiie execution of the click action can be deter- 
mined in real time by effecting the above-described 
pointing determination process, tiierefore. in combina- 
45 tion witii ttie process which is carried out in the informa- 
tion processing apparatus 14, the pointing 
determination process can be used in various forms of 
applications. 

[01 1 8] Fbr example, a tiiree dimensional image picked 
50 up from altitude showing a number of ships lined up is 
displayed on the display 12. In a state in which a user 
(the information inputting person 10) is pointing to a par- 
ticular ship within a virtual 3-D space represented by the 
tiiree dimensional image, when a specific pointing 
55 action (e.g.. a fbnvard click action) is candied out, data 
fbr the ship pointed to is displayed, and when another 
specific pointing action (e.g.. a backward click action) is 
executed, the aforementioned ship and any ottier ships 
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relevant to the selected ship are distinguished from the 
other shi|)s and thereby displayed. When the display is 
enlarged on the user's instruction (pointing action), the 
afblrementioned ship is enlarged and displayed so as to 
be at the center of the display Accordingly, it can be 5 
used as an interface for retrieving information from a 
ship database. Instead of ships, it should be appreci- 
ated that the present invention can be applied to auto- 
mobiles, aircraft, bicycles, electrical equipment, 
machine components or the like. ro 
[0119] Further, on the basis of CAD data created in 
the process of designing a building, 3-D inr^ge data 
which con'esponds to an image of the inside of the 
building while the building is still at the steel frame stage 
may be created and displayed on the display 12. The is 
display of the 3-D image may then be changed so that it 
appears to the user as if they are actually moving inside 
the building in response to their instruction. Then, if the 
designer (information inputting person) performs a spe- 
cific instruction (for example, a fonA/ard dick) while 20 
pointing to a specific point on the 3-D image displayed 
on the virtual 3-D space, the position pointed to may be 
stored as. for example, the starting point or terminal 
point of the plumbing layout. If another specific instruc- 
tion is made (for example, a backward click), then vari- 25 
ous types of plumbing layouts may be displayed 
together and the type of plumbing desired selected from 
these. The selected plumbing may then be displayed 
laid out between the starting point and terminal point of 
the plumbing layout overlapping the 3-D image. This 30 
would allow the layout of the of the plumbing to be veri- 
fied cubically The present invention would thus assist in 
the designing of the plumbing systems and the like of 
the plumbing systems and the like of the interior of a 
building or other construction by acting as a design sup- ss 
port system. 

[0120] As another example, the display 12 may be 
installed in the remote control room of a crane. A three 
dimensional image which is obtained from Images 
picked up of the working site by a plurality of video cam- 40 
eras which are mounted to the crane is displayed on the 
display 12. When an operator (an information inputting 
person) points to a specific position within a virtual 3-D 
space represented by a three dimensional image and 
provides a specific instruction (for example, a fonvard 4S 
click), then, for example, the crane's arm may be rotated 
and moved toward the specific position. When another 
specific instruction (for example, a backward click) is 
provided, for example, the aane*s arm may be moved 
vertically toward the specific position. Accordingly, the so 
present invention may be used for a three-dimensional 
operation system which enables an operator to operate 
a crane without having to climb to a great height. 
[0121] As another example, the three dimensional 
image representing a motor vehicle may be displayed ss 
on the display 12. If a user (the information inputting 
person) makes a specific instruction (for exarrple. a for- 
ward click) while pointing to the bonnet, for example, 



then the 3-D image may be changed to show the vehicle 
bonnet open. Then, if the user nmkes a further specific 
instruction (for example, a forward click) while pointing 
to the vehicle's engine, then the engine's performance 
and the like may be displayed overlapping the 3-D 
image. If another specific instruction is made (for exam- 
ple, a backward clid^. then that might be regarded as 
an instruction to cancel the previous instruction and the 
engine's performance and the like would be deleted. 
Thus the present invention might also be used for a vir- 
tual auto show room. When a specific instruction (for 
example, a fbnward click) is provided in a state in which 
a user (a information inputting person) is pointing at an 
engine of the vehicle, the engine performance or the like 
is displayed so as to overlap the three dimensional 
image. When another specific instruction (for example, 
a backward click) is provided, the instruction is deter- 
mined to be canceled, and the display of the engine per- 
formance or the like is canceled and can be used as a 
virtual showroom of the vehicle. In this case, for exam- 
ple, the action to change the display magnification 
which has been described above might be used as an 
action to change the orientation of the vehicle. For 
example, if the user made a zoom-in or zoom-out action 
while pointing to the front of the vehicle, then the entire 
vehicle might be shown as moving in either a clockwise 
or anti-ctockwise direction. 

[0122] The display 12, the hand pointing input appa- 
ratus 20. and the information processing apparatus 14 
might also, for example, be operated as a game 
machine which performs 3-D simulation games at 
amusement centers. 

[01 23] In the above descriptions, an exanrple in which 
two video cameras 36A and 36B are provided has been 
explained. However, the present invention is not limited 
to this, and more video cameras may be used to pick up 
images from each information inputting space, and to 
determine the instructions from the information inputting 
person 10. 

[0124] In the above description, the distance conver- 
sion conditions are set by considering a distance k 
between the reference point Pq and the characteristic 
point Px at the time when the information inputting per- 
son 10 points to a neutral point. However, the present 
invention is not limited to this. The distance conversion 
conditions can be set merely based on a maximum 
value k^ax ^^d a minimum value k^jn of the distance k. 
In this case, the conversion characteristics of the dis- 
tance conversion conditions can be made linear (for 
exannple. conversion characteristics in which the dis- 
tance L may vary in proportion to the distance k). How- 
ever, especially when a virtual 3-D space which is 
represented by a three dimensional image is a space 
whose depth is very long (for example, if a tiiree dimen- 
sional image which represents a part of the universe is 
displayed on the display means or the like), preferably, 
tiie conversion characteristics are made non-linear (for 
example, the conversion characteristics which may vary 
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the distance between the person to be recognized and 
the specific position in proportion to the number raised 
to the nth (n ^ 2) power of the change in the distance 
'4)etween the reference point and the characteristic 

point). 

[0125] In the above description, an example of an 
aspect in which an arbitrary position within a virtual 3-D 
space which is represented by a three dimensional 
image which is displayed on the display 12 has been 
explained. However, the object to which the information 
inputting person points is not limited to a virtual 3-D 
space. The present invention can be applied to cases in 
which the information inputting person points at an arbi- 
trary position within an actual 3-D space. The structure 
in which the information inputting person points at an 
arbitrary position within an actual 3-D space may be 
used in cases in which, for example, at a theater or the 
like, the illuminating direction of a spot light, the direc- 
tion of an acoustic beam which is formed by a large 
number of speakers which are lined up in an array is ori- 
ented in the direction in which an operator (information 
inputting person) is pointing. 
[0126] As desaibed above, in accordance with the 
first aspect of the present invention, a hand pointing 
apparatus extracts an image portion which con-esponds 
to the person to be recognized from a plurality of images 
which are obtained by picking up images of the state in 
which the person to be recognized is pointing at a spe- 
cific position within a 3-D space from a plurality of direc- 
tions, determines the 3-D coordinates of a characteristic 
point whose position may be changed by the person to 
be recognized bending or stretching an arm, and a ref- 
erence point whose position does not change even 
when the person to be recognized bends or stretches 
an arm, and thereby determines the direction in which 
the specific position exists within the 3-D space on the 
basis of the direction from the reference point to the 
characteristic point and determines the position of the 
specific position within the 3-D space along the depth 
direction thereof on the basis of the distance between 
the reference point and the characteristic point so that 
the 3-D coordinates of the specific position within the 3- 
D space can be determined. As a result, an excellent 
effect can be provided in which even when the informa- 
tion inputting person points at an arbitrary position 
within a 3-D space, the 3-D coordinates of the portion 
pointed to by the information inputting person can be 
determined. 

[0127] In accordance with the third aspect of the 
present invention, there is provided a hand pointing 
apparatus according to the second aspect of the 
present invention, wherein because the conversion 
characteristics of the conversion conditions which con- 
vert the distance between the reference point and the 
characteristic point into the distance between the per- 
son to be recognized and the specific position can be 
set on the basis of the width of the change in the dis- 
tance between the reference point and the characteris- 



tic point when the person to be recognized carries out 
the arm bending or stretching action, the conversion 
characteristics can be obtained in accordance with the 
individual physique of the person to be recognized in 

5 spite of the variation of the individual physique of the 
person to be recognized. As a result, an excellent effect 
can be obtained in which the location of the specific 
position in the depth direction of the specific position to 
which the person to be recognized points can be deter- 

10 mined accurately 

[0128] In accordance with the fourth aspect of the 
present invention, there is provided a hand pointing 
apparatus according to the first aspect of the present 
invention, wherein, in a state in which the hand of the 

15 person to be recognized is in a specific shape, and 
when the distance between the reference point and the 
characteristic point is changed, it is possible to control 
the three dimensional image displayed on the display 
means so as to be displayed in an enlarged/reduced 

20 state. In addition to the aforementioned effects, an 
excellent effect can be obtained in which the display 
magnification of the three dimensional image which is 
displayed on the display means can be changed reliably 
when the display nr\agnif ication of a three dimensional 

25 image is changed by the instruction from the person to 
be recognized. 

[0129] In accordance with the fifth aspect of tiie 
present invention, there is provided a hand pointing 
apparatus according to the first aspect of ttie present 

30 invention, because a predetermined process is exe- 
cuted when tiie detected speed of tiie change in the dis- 
tance between the reference point and tiie 
characteristic point is greater than or equal to a thresh- 
old value. In addition to tine aforementioned effects, an 

35 excellent effect can be obtained in which tiie action in 
which tiie person to be recognized instructs a predeter- 
mined action can be detected reliably 
[0130] In accordance with the sixtti aspect of tiie 
present invention, when the distance between the refer- 

40 ence point and tiie characteristic point increases at a 
speed which is greater than or equal to tiie tiireshold 
value, tiie first predetermined process can be executed, 
and when the distance between ttie reference point and 
tiie characteristic point decreases at a speed which is 

45 greater than or equal to the threshold value, the second 
predetermined process which is different from tiie first 
predetermined process can be executed. As a result, in 
addition to the aforementioned effects an excellent 
effect can be obtained in which tiie person to be recog- 

50 nized can select one of ttie first predetermined process 
and the second predetermined process and the process 
which is selected by ttie person to be recognized from 
ttie first predetermined process and the second prede- 
termined process can reliably be executed. 

55 [0131] In accordance witti tiie seventti aspect of tiie 
present invention, ttiere is provided a hand pointing 
apparatus according to the fifth aspect of the present 
invention, wherein a ttireshold value is set on the basis 
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Of the speed at which the distance between the refer- 
ence point and the characteristic point changes when 
the person to be recognized carries out the arm bending 
or stretching action of the person to be recognized. In 
addition to the aforementioned effects, an excellent 
effect can be obtained in that a threshold value can be 
determined in response to the physique, the muscular 
strength or the like of the individual person to be recog- 
nized, the action in which the person to be recognized 
instructs that a predetermined process be executed can 
be detected reliably so that a predetermined process 
can be executed in spite of the variation in the physique, 
the muscular strength or the like of each of the persons 
to be recognized. 

Claims 

1 . A hand pointing apparatus, comprising: 

image pickup means which picks up an image 
of a person to be recognized from a plurality of 
different directions; 

confuting means which extracts an image 
which con^esponds to the person to be recog- 
nized on the basis of a plurality of images 
obtained by said image pickup means which 
picks up images from a plurality of directions of 
the person to be recognized pointing to a spe- 
cific position within 3-D coordinates, and deter- 
mines the 3-D coordinates of a characteristic 
point whose position may be changed by said 
person to be recognized bending or stretching 
an arm, and a reference point whose position 
does not change even when said person to be 
recognized bends or stretches said arm; and 
determining means for determining the direc- 
tion in which the specific position exists within 
said 3-D space on the basis of the direction 
from said reference point to said characteristic 
point and for determining the location of said 
specific position within said 3-D space along 
the depth direction thereof on the basis of the 
distance between said reference point and said 
characteristic point, and thereby determines 
the 3-D coordinates of said specific position 
within said 3-D space. 

2. A hand pointing apparatus according to claim 1, 
wherein said determining means determines the 
location of said specific position within said 3-D 
space in the depth direction thereof by converting 
the distance between said reference point and said 
characteristic point into the distance between said 
person to be recognized and said specific position 
in accordance with predetermined conversion con- 
ditions. 

3. A hand pointing apparatus according to claim 2, fur- 



ther conprising: conversion conditions setting 
means which requests said person to be recog- 
nized to can^y out said arm bending or stretching 
action, and sets in advance said conversion condi- 

5 tions which convert the distance between said ref- 
erence point and said characteristic point into the 
distance between the person to be recognized and 
said specific position on the basis of the extent of 
the change in the distance between said reference 

10 point and said characteristic point when said per- 
son to be recognized carries out said arm bending 
or stretching action. 

4. A hand pointing apparatus according to claim 1 . fur- 
15 tiier conprising: 

display means which displays a three dimen- 
sional image; 

display control means which, displays said 
20 three dimensional image on said display 

means; and 

determining means which determines whether 
the hand of said person to be recognized is in a 
specific shape, 

25 wherein said three dimensional image is an 

image which represents a virtual 3-D space, 
and includes an image which is formed con- 
firming to a one-point perspective metiiod or 
two-point perspective method, an image which 

30 uses a liquid crystal shutter or a lenticular lens. 

and a stereographic image which is displayed 
by applying holographic technology, and said 
person to be recognized points to a specific 
position witiiin said virtual 3-D space which is 

35 represented by said ttiree dimensional image 

which is displayed on said display means, and 
in a state in which tiie hand of said person to be 
recognized is determined to be in a specific 
shape, when the distance between said refer- 

40 ence point and said characteristic point 

changes, said display control means controls 
said display means such that said tiiree dimen- 
sional image displayed on said display means 
is displayed so as to be enlarged or reduced 

45 according to ttie change in the distance 

between said reference point and said charac- 
teristic point. 

5. A hand pointing apparatus according to daim 1 , fur- 
so tiier comprising: processing means which detects 

tiie speed at which the distance between said refer- 
ence point and said characteristic point changes, 
and executes a predetermined process when tiie 
detected speed at which the distance between said 
55 reference point and said characteristic point 
changes is greater than or equal to a tiireshold 
value. 
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6. A hand pointing apparatus according to claim 5, 
wherein said processing means executes a first 
predetermined process when the distance between 
said reference point and said characteristic point 
increases at a speed which is greater than or equal s 
to said threshold value, and executes a second pre- 
determined process which is different from the first 
predetermined process when the distance between 
said reference point and said characteristic point 
decreases at a speed which Is greater than or equal w 
to said threshold value. 

7. A hand pointing apparatus according to claim 5, fur- 
ther comprising: threshold value setting means 
which requests said person to be recognized to is 
carry out the arm bending or stretching action to 
cause said processing means to execute said pre- 
determined process, and thereby sets said thresh- 
old value in advance on the basis of the speed at 
which the distance between said reference point 20 
and said characteristic point changes when said 
person to be recognized cames out the arm bend- 
ing or stretching action. 

8. A hand pointing apparatus according to claim 7, 25 
wherein said threshold value setting means sets, as 
said threshold value, a value which is less than the 
mean value of the speed at which said distance 
changes when said person to be recognized has 
earned out the arm bending or stretching action. 30 

9. A hand pointing apparatus according to claim 7, 
wherein said threshold value setting means sets, as 
said threshold value, the minimum value of the 
speed at which said distance changes when said 35 
person to be recognized canries out the arm bend- 
ing or stretching action. 

1 0. A hand pointing apparatus according to claim 3, fur- 
ther comprising: 40 



causes the person to be recognized to point to 
a single position which is marked by said 
marker which is positioned at the intermediate 
portion of said 3-D space along the depth 
direction thereof, and sets said conversion con- 
ditions on the basis of the distance between 
said reference point and said characteristic 
point at this time. 

1 1 . A hand pointing apparatus according to daim 3, fur- 
ther comprising: 

display means which displays a three dimen- 
sional image; 

display control means which displays said 
three dimensional image on said display 
means; and 

marker display means which displays a marker 
having an arbitrary shape which can be easily 
recognized by said person to be recognized, 
wherein said three dimensional image is an 
image which represents a virtual 3-D space, 
and includes an image which is formed con- 
forming to a one-point perspective method or 
two-point perspective method, an image which 
uses a liquid crystal shutter or a lenticular lens, 
and a stereographic image which is displayed 
by applying holographic technology, and 
said conversion conditions setting means 
causes said person to be recognized to point to 
a plurality of different positions which are 
marked by said marker along the depth direc- 
tion of said 3-D space, and sets said conver- 
sion conditions on the basis of the distance 
between said reference point and said charac- 
teristic point. 

12. A hand pointing apparatus according to claim 3. 
wherein said conversion conditions are conversion 
characteristics in which the distance between said 
person to be recognized and the specific position 
within said 3-D space which is pointed to by said 
person to be recognized changes linearly in 
response to the change in the distance between 
said reference point and said characteristic point at 
this time. 

13. A hand pointing apparatus according to claim 3, 
wherein said conversion conditions are conversion 
characteristics in which the distance between said 
person to be recognized and the specific position 
within said 3-D space which is pointed to by said 
person to be recognized changes non-tinearly in 
response to the change in the distance between 
said reference point and said characteristic point 

14. A hand pointing apparatus according to claim 4, 
wherein said display control means displays said 



display means which displays a three dimen- 
sional image; 

display control means which displays said 
three dimensional image on said display 45 
means; and 

marker display means which displays a marker 
having an arbitrary shape which can be easily 
recognized by said person to be recognized, 
wherein said three dimensional image is an so 
image which represents a virtual 3-D space, 
and includes an image which is formed con- 
forming to a one-point perspective method or 
two-point perspective method, an image which 
uses a liquid crystal shutter or a lenticular lens, ss 
and a stereographic image which is displayed 
by applying holographic technology, and 
said conversion conditions setting means 
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three dimensional image on said display means so 
as to.be enlarged or reduced by changing the dis- 
play magnification of said three dimensional image 
: which is displayed on said display means in 
response to the change in the distance between 5 
said reference point and said characteristic point. 

15. A hand pointing apparatus according to claim 3, 
comprising: 

10 

display means for displaying a three dimen- 
sional image: and 

display control means which displays said 
three dimensional image on said display 
means. is 
wherein said three dimensional image is an 
image which represents a virtual 3-D space, 
and includes an image which is formed con- 
forming to a one-point perspective method or 
two-point perspective method, an image which 20 
uses a liquid crystal shutter or a lenticular lens, 
and a stereographic image which is displayed 
by applying holographic technology, and 
said conversion conditions setting means cor- 
rects said conversion conditions in accordance 2S 
with the variation of the display magnification of 
said three dimensional image which is dis- 
played on said display means in an enlarged or 
reduced state. 

30 

16. A hand pointing apparatus according to claim 3. 
comprising: 

display means for displaying a three dimen- 
sional image; and 35 
display control means which displays said 
three dimensional image on said display 
means. 

wherein said three dimensional image is an 
image which represents a virtual 3-D space, 40 
and includes an image which is formed con- 
forming to a one-point perspective method or 
two-point perspective method, an image which 
uses a liquid crystal shutter or a lenticular lens, 
and a stereographic image which is displayed 45 
by applying holographic technology, and 
the display magnification of said three dimen- 
sional image which is displayed on said display 
means may vary linearly in response to the 
change in the distance between said reference so 
point and said characteristic point. 

17. A hand pointing apparatus according to claim 3, 
comprising: 

55 

display means for displaying a three dimen- 
sional image; and 

display control means which displays said 



three dimensional image on said display 
means, 

wherein said three dimensional image is an 
image which represents a virtual 3-D space, 
and includes an image which is formed con- 
forming to a one-point perspective method or 
two-point perspective method, an image which 
uses a liquid crystal shutter or a lenticular lens, 
and a stereographic image which is displayed 
by applying holographic technology, and 
the display magnification of said three dimen- 
sional image which is displayed on said display 
means may vary non-linearly in response to the 
change in the distance between said reference 
point and said characteristic point. 

18. A hand pointing apparatus according to claim 7. 
wherein said threshold value setting means sets 
said threshold value anew each time a person to be 
recognized is replaced by anotiier person to be rec- 
ognized. 
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